System and methods for feedback-based data transmission

ABSTRACT

A computer-implemented method includes receiving a feedback message from a receiver of image data. The feedback message includes a reference frame identifier associated with a reference frame correctly received or decoded by the receiver. The method further includes determining a difference between the reference frame identifier and a current frame identifier. The current frame identifier is associated with a current frame to be encoded and the difference indicates a number of frames between the reference frame and the current frame. The method also includes encoding the current frame based at least in part on the difference.

CROSS-REFERENCE

This application is a continuation of application Ser. No. 16/457,809,filed Jun. 28, 2019, which is a continuation of InternationalApplication No. PCT/CN2017/120225, filed Dec. 29, 2017, which claims thebenefit of priority to International Application No. PCT/CN2016/113944,filed Dec. 30, 2016, and International Application No.PCT/CN2016/113899, filed Dec. 30, 2016, the entire contents of all ofwhich are incorporated herein by reference.

BACKGROUND

Unmanned aerial vehicles (UAVs) equipped with imaging devices are widelyused to perform a variety of tasks such as surveillance and tracking,remote sensing, search and rescue, scientific research, and the like.However, low-latency transmission of image data such as videos and stillimages remains challenging in an environment with wireless and/orunreliable communication channels. In such an environment, data loss ordecoding error of image frames can occur frequently, leading todeterioration of image quality and delay. A traditional approach forerror recovery is to have a transmitter of data (e.g., UAV) periodicallytransmit intra-coded frames (or I-frames), so that a receiver of thedata (e.g., a remote terminal) does not need to rely on any otherpotentially missing or corrupt frames for decoding. However, in somecases, such an error recovery approach may not be timely enough tosatisfy the requirement of low-latency system. Additionally, such anapproach can increase the bitrate required to encode the frames, lowercoding efficiency, and lead to inefficient channel usage.

SUMMARY

According to embodiments, a method is provided comprising receiving,from a receiver of image data, a feedback message comprising feedbackinformation, obtaining a current frame to be encoded, encoding thecurrent frame based at least in part on the feedback information, andtransmitting the encoded current frame to the receiver.

According to embodiments, a method is provided comprising determining,at a receiver configured to receive image data transmitted from atransmitter, a receiving status with respect to the receiver, generatingfeedback information based at least in part on the receiving status, andtransmitting the feedback information to the transmitter.

It shall be understood that different aspects of the disclosure can beappreciated individually, collectively, or in combination with eachother. Various aspects of the disclosure described herein may be appliedto any of the particular applications set forth below or datacommunication between any other types of movable and/or stationaryobjects.

Other objects and features of the present disclosure will becomeapparent by a review of the specification, claims, and appended figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 illustrates an exemplary system for implementing feedback-baseddata transmission, in accordance with embodiments.

FIG. 2 illustrates an exemplary process for communication between atransmitter and a receiver.

FIG. 3 illustrates an exemplary process for data transmission, inaccordance with embodiments.

FIG. 4 illustrates exemplary processes for receiving data, in accordancewith embodiments.

FIG. 5 illustrates exemplary processes for receiving data, in accordancewith embodiments.

FIG. 6 illustrates another exemplary process for receiving data, inaccordance with embodiments.

FIG. 7 illustrates another exemplary process for receiving data, inaccordance with embodiments.

FIG. 8 illustrates another exemplary process for transmitting data, inaccordance with embodiments.

FIG. 9 illustrates an exemplary process for managing a reference framelist, in accordance with embodiments.

FIG. 10 illustrates another exemplary process for transmitting data, inaccordance with embodiments.

FIG. 11 illustrates exemplary processes for encoding a frame, inaccordance with embodiments.

FIG. 12 illustrates a movable object including a carrier and a payload,in accordance with embodiments.

FIG. 13 is a schematic illustration by way of block diagram of a systemfor controlling a movable object, in accordance with embodiments.

DETAILED DESCRIPTION

The systems, devices, and methods are provided for low-latency datatransmission. In particular, a data transmitting terminal (also referredto as a transmitting terminal or transmitter) can be configured toencode frames based feedback information received from a data receivingterminal (also referred to as a receiving terminal or receiver). Thefeedback information can indicate a status with respect to datatransmission at the receiver. For example, the feedback information mayindicate that a received frame cannot be decoded successfully (e.g., dueto transmission error or decoder malfunction). The feedback informationcan also include information that may be helpful for the transmitter todetermine how to effectively recover from an error. For example, thefeedback information may also include an identifier (e.g., a uniquesequence number) of a frame that has been previously received and/orsuccessfully decoded by the receiver. Depending on the feedbackinformation, the transmitter can choose a suitable way to encode acurrent frame. For example, when the feedback information indicates anerror, the transmitter can choose a suitable error recovery mechanismbased on the feedback information and/or the current state at thetransmitter. For example, the transmitter may determine whether toencode the current frame under an intraframe coding mode or aninterframe coding mode. The transmitter may determine which referenceframe to select to encode the current frame under the interframe codingmode.

Advantageously, the techniques described herein address some or all ofthe problems associated with existing techniques. In particular, thefeedback information provided by the receiver allows the transmitter toencode frames in such a way as to respond effectively, efficiently, andtimely to any errors without significant impact on performance andchannel usage. For instance, instead of periodically transmittingintraframes as is done in traditional techniques, the transmitter can beconfigured to transmit the more efficiently encoded interframes most ofthe time and only transmit intraframes under limited circumstances, asguided by the feedback information.

FIG. 1 illustrates an exemplary system 100 for implementing feedbackbased data transmission, in accordance with embodiments. The system 100can comprise a transmitting terminal (also referred to as a transmitter)102 and a receiving terminal (also referred to as a receiver) 104 thatcommunicate with each other over one or more communication channels (notshown). The transmitter 102 can be configured to encode (e.g., compress)image frames 106, such as video frames and still image frames, and totransmit the resulting encoded data 108 to the receiver 104. Thereceiver 104 can be configured to decode (e.g., decompress) the encodeddata 108 to generate reconstructed frames 124, for example, for playbackor display purposes. The receiver 104 can also be configured to generatefeedback information 122 based on the receiving status at the receiver104, for example, with respect to the encoded data 108 and to transmitthe feedback information 122 to the transmitter 102. The transmitter 102can be configured to encode frames 106 based on the feedback information122 from the receiver 104. In some embodiments, the communicationchannel used for transmitting the feedback information 122 (alsoreferred to as the feedback channel) may or may not be the same as thecommunication channel used for transmitting the encoded data 108 (alsoreferred to as the data channel).

In some embodiments, the transmitter 102 can be implemented on a movableobject described herein (e.g., a UAV) and configured to receive theimage frames 106 from one or more image capturing or image acquisitiondevices or systems such as cameras or video cameras, vision sensors, andthe like, carried by such movable objects. The transmitter 102 may be apart of an image acquisition system. For example, the transmitter 102and an image acquisition module may be enclosed in the same housing.Alternatively, at least a portion of the transmitter 102 may beseparately located from the image acquisition device. For example, oneor more components of the transmitter 102 may be located outside ahousing for the image acquisition device. In some embodiments, thetransmitter may include or be included in an image processing system.

In some embodiments, the transmitter 102 can comprise a feedbackprocessing module 110, a mode selection module 112, an encoder 114, apackaging module 116, a rate control module (or rate controller) 118,and a transmitter context management module 120. Optionally, thetransmitter 102 may be operably connected to or include a communicationunit (not shown) configured to communicate with the receiver 104. Thesecomponents can be implemented by one or more processors configured toimplement executable instructions stored in non-transitory storagemedia. The one or more processors can include ARM processors,field-programmable gate arrays (FPGAs), application-specific integratedcircuit (ASIC), central processing units (CPUs), graphics processingunits (GPUs), and the like. In some embodiments, the components of thetransmitter 102 may be implemented using hardware accelerationtechniques.

The feedback processing module 110 can be configured to receive andprocess feedback information from the receiver. For instance, thefeedback processing module 110 can be configured to extract data such asa status indicator and a frame identifier from a feedback message. Theextracted data may be processed to determine whether an error has beendetected at the receiver.

The mode selection module 112 can be configured to determine a mode forencoding a current frame (also referred to as an coding mode or a codingmode). In some embodiments, the mode selection module 112 can beconfigured to select from a plurality of coding modes comprising anintraframe coding mode (also referred to as “intraframe coding mode” or“intraframe mode”), an interframe coding mode using a short-termreference frame (also referred to as a “short-term interframe codingmode” or “short-term interframe mode”), and an interframe coding modeusing a long-term reference frame (also referred to as a “long-terminterframe coding mode” or long-term interframe mode”).

Under an intraframe coding mode, a current frame is encoded byexploiting spatial redundancy, e.g., correlation among pixels, withinthe current frame. An intraframe coded frame is also referred to as anI-frame. For example, prediction values may be calculated byextrapolation from already coded pixels. An I-frame is encoded spatiallywithout reference to any other frames. Under an interframe coding mode,a current frame is encoded by exploiting temporal redundancy betweenneighboring frames. For example, a current frame (also referred to as aP-frame or target frame) can be predicted from a reference frame thatprecedes it in a sequence of frames. The reference frame can itself be aP-frame or an I-frame. In some cases, the target frame may be intraframeencoded as well as interframe encoded. In other cases, the target framemay only be interframe encoded. A reference frame that immediatelyprecedes the target frame is referred to as a short-term referenceframe. Interframe encoding using a short-term reference frame isreferred to as short-term interframe encoding or short-term interframecoding. A reference frame that does not immediately precede the targetframe is referred to as a long-term reference frame. Interframe encodingusing a long-term reference frame is referred to as long-term interframeencoding or long-term interframe coding.

The mode selection module 112 can be configured to determine a codingmode based at least in part on the feedback information 122. Forexample, the mode selection module 112 can be configured to receive theprocessing result from the feedback processing module 110. In some otherembodiments, the feedback processing module 110 may be omitted and themode selection module 112 may be configured to process the feedbackinformation 122 directly.

In some embodiments, mode selection may be based at least in part on adifference between an identifier of the current frame to be encode and areference frame identifier included in the feedback information. If thedifference between the identifiers is less than or equal to apredetermined threshold value, then a short-term interframe mode may beselected. In other words, the current frame is interframe encoded usinga short-term reference frame. Otherwise, an intraframe mode or along-term interframe mode may be selected.

Additionally or alternatively, the mode selection may be based at leastin part on transmitter context information. The transmitter contextinformation may be managed by the transmitter context management module120. In various embodiments, the transmitter context information caninclude interframe coding information, intraframe coding information,coding parameters, and any other state information related to thetransmitter. For example, the context information may include areference frame list that includes reference frames that are useful forencoding purposes. In some embodiments, the reference frame list can beimplemented using a first-in-first-out (FIFO) queue, where a newlyencoded frame is pushed to the head of the queue and the reference frameat the end of the queue is removed when a new frame is added. In variousembodiments, any other suitable data structures can be used to implementthe reference frame list, including a stack, a linked list, a table, ahierarchy, a tree, and the like.

In some embodiments, a sliding window mechanism can be used to controlthe size of the reference frame list. The sliding window can be updated(e.g., size increase/decrease, advanced forward) based on a variety offactors discussed herein. For example, the reference frame list may becleared (e.g., setting the size of the sliding window to 0), whenintraframe coding is required. As another example, after encoding a newframe, the reference frame list may be advanced forward to include thenewly encoded frame. Frames at the end of the list may be removed tokeep the size of the list less than or equal to a predetermined maximumsize (e.g., 8 or 16).

In some embodiments, the head of the reference frame list is always usedfor encoding a current frame. In such embodiments, the reference framelist may be updated to insert or remove certain frames, so that the headof the list is the correct reference frame to use for encoding. Forexample, a long-term reference frame may be moved to the head of thelist so it may be used for encoding the current frame. In otherembodiments, the head of the reference frame list is not always used forencoding the current frame. In such embodiments, a pointer may be keptand updated to point to the reference frame which will be used forencoding the current frame. In some embodiments, an empty referenceframe list means that the current frame will be intraframe encoded.

Optionally, the mode selection may be based at least in part oninformation provided by the rate control module 118. For example, therate control module 118 may be configured to control a bitrate of theencoded image stream (e.g., by changing coding parameters such asquantization parameter), such that the resulting bitrate or resolutionmatches a current or expected channel condition or a specific usescenario. In some cases, a change in coding parameters from the ratecontrol module indicates a change in bitrate, which requires selectionof the intraframe coding mode.

The encoder 114 can be configured to encode a current frame (e.g.,obtained from an image acquisition unit), based on the coding modeselected by the mode selection module 112 and/or the coding parametersprovided by the rate control module 118.

Generally, the encoder is configured to receive input data (such as theimage frames 106), encode the input data (such as the image frame 106),and provide output data comprising the encoded input data. The inputdata can include text, images, graphic objects, animation sequences,audio recordings, videos, or any other data that needs to be encoded. Insome cases, the input data may include sensing data from one or moresensors such as vision sensors (e.g., cameras, infrared sensors),microphones, proximity sensors (e.g., ultrasound, lidar), positionsensors, temperature sensors, touch sensors, and the like. In somecases, the input data may include input from a user such as biometricinformation including facial features, fingerprint scan, retina scan,voice recording, DNA samples, and the like.

Encoding of the input data may be necessary for efficient and/or securetransmission or storage of the data. Encoding of the input data caninvolve data compression, encryption, error encoding, format conversion,and the like. For example, multimedia data such as video or audio may becompressed to reduce the number of bits that are transmitted over thenetwork. Sensitive data such as financial information and personallyidentifying information may be encrypted before being transmitted orstored to protect confidentiality and/or privacy.

Any suitable encoding techniques may be used to encode input data. Thetype of encoding applied may depend on the data to be encoded andspecific encoding requirements. For example, lossy compressiontechniques (e.g., transform coding) may be used to encode multimediadata such as audio signals and image data (e.g., still images or video);whereas lossless compression techniques may be used to encode text andbinary data (e.g., executable files). Audio signals may be compressedusing audio compression methods and video data may be compressed usingvideo compression methods. In some cases, the encoding of certain inputdata (e.g., financial data, medical or personal information) may begoverned by established laws, regulations, or standards. Furthermore,available computing resources (e.g., CPU, memory, buffer capacity)and/or network environment (e.g., bandwidth, security) may affect thetype of encoding that is performed. For instance, a server computerequipped with powerful CPUs may be configured to implement a relativelycomplex encoding algorithm, whereas a mobile device with limitedcomputing resources may be configured to implement a relatively simpleencoding algorithm. As another example, if the encoded data will betransmitted in a relatively secure environment (e.g., over a private orfirewall-protected network), then the input data may be encrypted withan encryption algorithm that has a relatively low security level. On theother hand, if the encoded data will be transmitted in a less secureenvironment (e.g., over the Internet), then the input data may beencrypted with an encryption algorithm that has a relatively highsecurity level.

In some embodiments, the encoder 114 may implement one or more differentcodecs. Each of the one or more codes may comprise codes, instructionsor computer programs that implement different encoding algorithms. Asuitable codec may be selected to encode a given set of input data basedon various factors such as discussed above, including the types and/orsources of input data, the receiving entities of the encoded data,availability of computing resources, network environment, businessrequirements, regulations and standards, and the like.

In an example, an encoder may be configured to encode a series of videoframes. Data in each frame may be encoded using a series of steps. Theencoding may be based entirely on spatial information contained within aframe (e.g., an intra frame or I frame). In such embodiments, theencoding steps may include a transform step, a quantization step, and anentropy encoding step. During the transform step, the raw input data istransformed to a different domain (e.g., spatial frequency domain)suitable for the data content of the input data (e.g., video). Anysuitable transform coding techniques may be used including Fourier-typetransforms such as discrete cosine transform (DCT) or modified DCT.Using DCT as an example, a DCT matrix is determined based on a size ofthe data unit. The data unit may include a block of 4×4 or 8×8 pixels, amacroblock of 16×16 pixels, or any suitable set of data. The DCT matrixis then applied to the data unit using matrix multiplication, yielding atransformed matrix comprising transform coefficients.

In the quantization step, the coefficients in the transformed matrix maybe quantized, for example, by dividing each coefficient by acorresponding element in a quantization matrix, and then rounding to thenearest integer value. The quantization matrix may be derived using aquantization parameter (also referred to as a quantization index). Forexample, the quantization parameter may be the value for each element ofthe quantization matrix. As another example, some or all of the elementsin the quantization matrix may be multiplied (scaled) by thequantization parameter and the scaled quantization matrix may be used toquantize the transformed matrix. The quantization parameter may be aninteger within a certain range (e.g., between and including 0 and 128).Typically, the higher the value of the quantization parameter, thelarger the quantization step size is and the larger the element valuesare in the quantization matrix. This causes more transform coefficientsto be quantized to zero or near-zero, which means that less bits wouldbe required to encode the quantized coefficients. The more zero ornear-zero coefficients there are, the less bits are required to encodethe coefficients, resulting in lower bit size (and hence lower bitrate)for the data unit represented by the coefficients. The opposite is alsotrue, that is, a lower value of a quantization parameter corresponds toa smaller quantization step size, a greater number of bits required toencode the quantized coefficients, and a higher bit size (and hencehigher bitrate) for the data unit encoded using the quantizationparameter. Techniques are provided herein for controlling the bitrate ofthe encoded input data by varying the quantization parameters used toencode portions of the input data.

In the entropy encoding step, the quantized coefficients in a quantizedmatrix are scanned in a predetermined order and encoded using anysuitable coding technique. Since most of the non-zero DCT coefficientsare likely concentrated in the upper left-hand corner of the matrix, azigzag scanning pattern from the upper left to the lower right istypical. Alternative scanning order such as a raster scan may be used.The scanning order may be used to maximize the probability of achievinglong runs of consecutive zero coefficients. The scanned coefficients canthen be encoded using run-length encoding, variable-length encoding, orany other entropy encoding techniques, to generate the output data.

The above-described encoding process can be used to encode intra frames(I frames) based primary on spatial information contained within theintra frames. Additionally or alternatively, the encoder may beconfigured to exploit temporal redundancy between the frames and encodeinter frames (e.g., P frames or B frames) based on forward and/orbackward predictions made from previous or subsequent frames. In suchembodiments, the encoding steps may include a prediction step, atransform step, a quantization step, and an entropy encoding step. Inthe prediction step, a frame may be forward and/or backward predictedbased on a previous frame and/or a subsequent frame by estimating motionof the camera and/or objects in the video. Any suitable motionestimation techniques may be used to determine the motion vectorsbetween adjacent frames, including pixel based methods (e.g.,block-matching) and feather based methods (e.g., corner detection). Ifan acceptable match of a corresponding data unit (e.g., macroblock) isnot found, then the encoder may encode the data unit as an intra dataunit, as described above. Otherwise, the predicted frame may besubtracted from its reference and the residual error frame may begenerated. The data included in the residual error frame may bespatially encoded in a similar fashion as for an intra frame, asdescribed above. For example, one or more data matrices of the residualerror frame may be transformed (e.g., using DCT) and quantized. Thequantized transform coefficients of the residual error frame, the motionvectors or the difference between motion vectors of adjacent frames,along with any other suitable data needed to reconstruct the frame maybe entropy encoded. As in intra encoding, the bitrate of the encodeddata may be controlled at least in part by a quantization parameterprovided by a rate controller.

The rate control module 118 (also referred to as the rate controller)can be configured to control a bitrate of the output encoded data bycontrolling the coding parameters (also referred to as rate controlparameters) used by the encoder 114. The bitrate may be controlled to bewithin a certain range (e.g., no more than a maximum bitrate, no lessthan a minimum bitrate) or close to a target average bitrate.Alternatively, the bitrate may be controlled to vary depending on thecomplexity of the frames, bandwidth limit, buffer capacity, and otherfactors. In some cases, rate control may be enforced at one or morelevels such as groups of pictures (GOP) level, frame level, slice level,macroblock level, block level, pixel level, and the like.

The coding parameters can include one or more quantization parametersfor controlling the quantization step of the encoding process and hencethe bitrate of the resulting output data. The quantization parametersmay include, for example, a quantization step size, a value indicativeof or related to a quantization step size such as a QP used in H.264 orsimilar encoders, a quantization matrix or a reference thereof, and thelike. The coding parameters may include parameters for controlling otheraspects of the encoding process such as the prediction step, thetransform step, and/or the entropy encoding step. For instance, codingparameters may include a cutoff index used for removing certain highfrequency coefficients before the coefficients are entropy encoded.Other examples of the coding parameters may include bit allocationinformation (e.g., maximum, minimum, or target bits allocated forencoding a data unit), a frame rate, a size of a data unit to betransformed and quantized, motion detection thresholds used to determinewhether to code or skip coding a data unit (e.g., macroblock), Lagrangemultiplier used in rate distortion optimization, algorithms andparameters used for the prediction, transform, and/or entropy encodingsteps, and the like.

The rate controller may be configured to control rate (e.g., byproviding the code parameters) based at least in part on outputinformation about the output data from the encoder. The outputinformation may be provided by the encoder or optionally derived by therate controller based on the output data. The output information mayinclude, for example, a number of bits used to encode a data unit (e.g.,a frame, a slice, a macroblock), parameters (including algorithms) usedto encode the data unit, encoder resource information (e.g., CPU/memoryusage, buffer usage), and the like. Such information may be used by therate controller to adjust one or more coding parameters (e.g., aquantization parameter) for one or more subsequent data units.

The rate controller may optionally be configured to control rate basedat least in part on input information about the input frames. Inputinformation may include any characteristics of the input data that maybe useful for rate control, such as resolution, size, image complexity,texture, luminance, chrominance, motion information, and the like. Forexample, highly complex input data may be encoded with a higher bitratethan less complex input data.

In some embodiments, the rate controller may be configured to controlrate based on one or more rate control threshold parameters. The valuesof the threshold parameters may be predefined and/or dynamically updatedby a user, a system administrator, the rate controller, or any othercomponent or device. The rate control threshold parameters may be usedto derive coding parameters. In some embodiments, the threshold valuesused to determine the coding parameters for encoding a given slice mayvary depending on an encoding order of the slice relative to otherslices of a frame.

In some embodiments, the rate controller 118 may be configured tocontrol rate based on additional information. Such information mayinclude decoder information from an entity configured to receive,decode, and/or playback or display the output data (e.g., decoder bufferusage, delay, noise, playback quality), current computing environment(e.g., network bandwidth, workload), user instructions, or any othersuitable information relevant to rate control.

Still referring to FIG. 1, the packaging module 116 of the transmitter102 can be configured to package the encoded data from the encoder 114for transmission to the receiver 104. For example, the packaging module116 can be configured to package the encoded data into one or morepackets. Each packets may comprise at least a portion of the encodeddata that is encapsulated by metadata about the encoded data such as aheader portion (affixed before the encoded data) or a tail portion(appended to the end of the encoded data). Such metadata may beconfigured to facilitate data verification, error detection, and/ordecoding at the receiver. The packaging module 116 may pass the packetsonto a communication module (not shown), which transmits the packets tothe receiver 104. Alternatively, the packaging module 116 may beconfigured to transmit the packets.

In some embodiments, the receiver 104 can be implemented on a remoteterminal described herein (e.g., a remote controller, a mobile device)configured to receive the encoded image frames 108 from the transmitter102. In some embodiments, the receiver 104 can comprise a verificationmodule 128, a feedback generation module 130, a decoder 132, and areceiver context management module 134. Optionally, the receiver 104 maybe operably connected to or include a communication unit (not shown)configured to communicate with the transmitter 102. These components canbe implemented by one or more processor configured to implementexecutable instructions stored in non-transitory storage media. The oneor more processors can include ARM processors, ASICs, FPGAs, CPUs, GPUs,and the like. In some embodiments, the components of the receiver 104may be implemented using hardware acceleration techniques.

The verification module 128 can be configured to verify the integrity ofthe encoded data 108. For example, the verification module 128 may beconfigured to receive packets containing encoded data 108 from thetransmitter 102, extract the information contained in the packets, andverify that no error occurred during transmission. For example, theverification module 128 may be configured to extract metadata (e.g.,header and/or tail information) associated with the encoded data fromeach packet. The metadata may include verification data useful forverifying integrity and/or authenticity of the encoded data. Examples ofsuch verification data can include a error-detection code orerror-correction code such as a checksum, a length, digital signature,encryption key information, timestamp, or any other verification data.If encoded data is transmitted in multiple packets, then each of thepacket must be received and verified to detect any transmission error.Then the encoded data extracted from each of the packets may be combinedin a given order (e.g., as determined by a packet sequence numberassociated with each packet) before the combined data is decoded by adecoder.

In some embodiments, the encoded data can be checked directly using theverification data. Alternatively or additionally, the encoded data canbe transformed and/or decoded (e.g., uncompressed, decrypted) beforebeing checked using the verification data. Such verification may be usedto determine whether an error or security breach occurred during datatransmission (e.g., data loss or data modification). Alternatively oradditionally, such verification may be used to determine whether anerror occurred locally at the receiver, e.g., during the decodingprocess.

Examples of the metadata can include frame identifier, frame type (e.g.,I-frame, short-term-P-frame, long-term-P-frame), reference frame offset,timestamp, frame rate, data rate, resolution, and the like. The frametype can correspond to a mode under which the frame is encoded. Forexample, the frame type can be an I-frame or a P-frame. In anotherexample, the frame type may be an I-frame, a short-term-P-frameinterframe encoded using a short-term reference frame, or along-term-P-frame interframe encoded using a long-term reference frame.In some embodiments, the metadata about the encoded frame can be used todetermine whether a given frame can be reconstructed without having todecode the encoded data. For example, the frame identifier and frametype can be verified to ensure that the correct frame is received. Thereference frame offset can be used to determine availability of areference frame useful for decoding the received frame. If such areference frame is not received and/or decoded correctly, then it can bedetermined that the received frame cannot be reconstructed successfullywithout decoding the received frame. In some embodiments, some or all ofthe metadata may be used to facilitate the decoding of the encoded data.For example, the frame type can be used to determine the correspondingdecoding mode and the reference frame index can be used to locate thereference frame used to decode the encoded frame. As another example,the frame rate and the data rate can be used to dictate aspects of thedecoding process.

In some embodiments, the verification module 128 can be configured todetermine an operational status of one or more software and/or hardwarecomponents of the receiver. As discussed above, certain verificationdata received with the encoded data can be used to verify whether thedecoder is working properly. The verification module 128 can beconfigured to indicate a hardware and/or software failures (e.g., buffererrors, device failure, memory failure, operating system error,application error). The verification module 128 can be configured todetermine the status based on information (e.g., error code) providedsuch components or by polling such components (e.g., on a regular basisand/or under a diagnostic mode).

The result of the verification module 128 can be provided to thefeedback generation module 130 and/or the decoder 132. The feedbackgeneration module 130 can be configured to generate feedback informationregarding the receiver based on the result of the verification module128 and/or based on information provided by the decoder 132. Forexample, the result from the verification module 128 may indicatewhether an error has occurred during transmission of a frame (e.g., dataloss or data modification). Information provided by the decoder 132 mayindicate whether an error has occurred during decoding of a frame. Invarious embodiments, the feedback generation module 130 may beconfigured to generate the feedback information based on the result ofthe verification module 128 alone, information provided by the decoder132 alone, or both. In some embodiments, the feedback generation module130 may be configured to generate feedback information based on otherinformation such as a status of a hardware/software component of thereceiver, an environment of the receiver (e.g., noise or interference),a physical position/orientation of the receiver, a user input, and thelike.

In some embodiments, the feedback information may include informationabout a status indicator and a frame identifier. The status indicatormay indicate whether a certain frame has been received successfully. Thestatus indicator may indicate whether a certain frame has been decodedor reconstructed successfully. As used herein, reconstructing a framecomprises decoding all or portions an encoded frame. Reconstructing aframe may optionally include combining portions of encoded frame beforethe decoding process, and/or combining decoded portions of the frameafter the decoding process. The status indicator may indicate whether ahardware/software component of the receiver is working properly. Theframe identifier may indicate a frame for which the status indicator isrelevant. For example, if the status indicator indicates an error, theframe identifier may identify the last frame or any previous frame thathas been correctly received and/or successfully decoded before thegeneration of the feedback information.

In some embodiments, the feedback information may include otherinformation about the receiver that may be useful for improving theefficiency and reliability of data transmission. Such information mayinclude, for example, a timestamp associated with the generation and/ortransmission of the feedback information, an interval at which feedbackinformation is being sent, synchronization information useful forsynchronizing the data transmission between transmitter and the receiver(e.g., working frequency or frequencies, hopset information), and thelike.

In some embodiments, the feedback information may be generated at afixed feedback generation interval or at a variable feedback generationinterval. The interval may be previously-agreed on by the transmitterand the receiver. The interval may be dynamically determined based atleast in part on channel capacity, the requirement of a transmitterapplication regarding the promptness of error recovery, bitraterequirement, error rate (e.g., bit error rate or bit error ratio) withina predetermined period of time, statistics regarding previous datatransmissions, and the like. Alternatively or additionally, thegeneration of feedback information may be triggered by certainpredefined events such as a change in an environment of the receiver(e.g., a change in signal to noise ratio (SNR), a change in channelcapacity or network bandwidth), a change in a hardware or softwarestatus or setting, after receiving or decoding of a new frame, and thelike.

In some embodiments, the feedback information may be provided to acommunication unit (not shown), which may be configured to transmittedto the feedback information to the transmitter in a feedback message ata fixed feedback transmission interval or at a variable feedbacktransmission interval. The interval may be previously-agreed upon by thetransmitter and the receiver. The interval may be dynamically determinedbased on channel capacity, bitrate requirement, information provided bythe transmitter, and the like. For instance, the receiver may beconfigured to transmit at a first interval. The transmitter may beconfigured to transmit a second interval to the receiver. The receivermay be configured to transmit at a third interval that is determinedbased on the first interval and the second interval. For example, thethird interval may be a maximum, a minimum, or an average of the firstand the second intervals.

Alternatively or additionally, the transmission of feedback informationmay be triggered by certain predefined events such as the generation ofthe feedback information, a change in an environment of the receiver(e.g., a change in signal to noise ratio (SNR), a change in channelcapacity or network bandwidth), a change in a hardware or softwarestatus or setting (e.g., detection of a hardware or software failure orerror), after receiving or decoding of a frame, when the transmissionerror rate has exceeded a predetermined threshold value, and the like.In some examples, the feedback information may be transmitted only aftera predetermined threshold amount of time has lapsed since the generationof the feedback information. In some other examples, the feedbackinformation is transmitted within a predetermined threshold amount oftime since the occurrence of certain events (e.g., detection of anerror). In various embodiments, the intervals and/or thresholdsdiscussed herein (e.g., with respect to feedback generation and/orfeedback transmission) may be predefined or dynamically configurable bya user, or an automatic or semi-automatic process. In some examples, theautomatic or semi-automatic process may be based on machine-learningtechniques.

In some embodiments, the decoder 132 can be configured to decodereceived data in order to reconstruct a frame. A decoder may beconfigured to perform decoding steps that are the inverse of theencoding steps of the encoder described herein to generate decoded orreconstructed data. The reconstructed data may then be displayed orplayed back. For example, to decode intra encoded data (e.g., I frames),the decoding steps include an entropy decoding step (e.g., usingvariable length decoding), an inverse quantization step, and an inversetransform step (e.g., using Inverse Discrete Cosine Transform (IDCT))that perform the inverse of the corresponding entropy encoding,quantization, and transform steps of the encoder. To decode interencoded data (e.g., B frames or P frames), the decoding process caninclude additional motion compensation support.

As discussed herein, aspects of the decoding process may be controlledby information provided by the verification module. For example,metadata associated with the encoded data may be used to determine acode mode, a reference frame to use, if any, and/or one or more codingparameters for decoding a given frame.

In some embodiments, the receiver context management module 134 can beconfigured to manage receiver context information. Receiver contextinformation can include information useful for decoding encoded framesreceived from the transmitter. For example, the context information mayinclude an identifier of the last successfully decoded frame. Thisinformation may be useful for generating feedback information. In someembodiments, the receiver context management module 134 can maintain areference frame list that includes one or more frames that have beensuccessfully received and decoded by the receiver. The reference framelist may be used by the decoder to decode received encoded frames. Insome cases, the reference frame list at receiver (also referred to as“receiver reference frame list”) may include the same as or a subset ofthe reference frame list at the transmitter (also referred to as“transmitter reference frame list”). In some other cases, the referenceframe list at the receiver may include one or more frames that are noton the reference frame list at the transmitter. In some embodiments, thereceiver reference frame list may be managed in a similar manner as thetransmitter reference frame list described above. For example, a slidingwindow mechanism may be used to control a size of the reference framelist. The sliding window may be “advanced” to include a newly decodeframe and/or updated to change the order of the reference frames asdescribed herein.

FIG. 2 illustrates an exemplary process 200 for communication between atransmitter 202 and a receiver 204. The transmitter 202 and the receiver204 may be similar to the transmitter 102 and the receiver 104 of FIG.1, respectively. Some or all aspects of the process 200 (or any otherprocesses described herein, or variations and/or combinations thereof)may be performed by one or more processors onboard the UAV, a payload ofthe UAV (e.g., an imaging device), and/or a remote terminal. Some or allaspects of the process 200 (or any other processes described herein, orvariations and/or combinations thereof) may be performed under thecontrol of one or more computer/control systems configured withexecutable instructions and may be implemented as code (e.g., executableinstructions, one or more computer programs or one or more applications)executing collectively on one or more processors, by hardware orcombinations thereof. The code may be stored on a computer-readablestorage medium, for example, in the form of a computer programcomprising a plurality of instructions executable by one or moreprocessors. The computer-readable storage medium may be non-transitory.The order in which the operations are described is not intended to beconstrued as a limitation, and any number of the described operationsmay be combined in any order and/or in parallel to implement theprocesses.

In the illustrated embodiment, the transmitter 202 can be configured totransmit 206 an encoded frame to the receiver 204. The encoded frame mayor may not be enclosed in additional header/tail information that isadded to facilitate verification and/or decoding of the encoded data. Insome embodiments, the encoded frame may be encoded based at least inpart on feedback information transmitted by the receiver 204.

Subsequently, the receiver 204 can be configured to generate 208feedback information. The feedback information may be generated at leastpartially in response to receiving of the encoded frame. For example,the receiver 204 may have received the encoded data 206, but hasdetected an error in verification of the received data. In this case,feedback information indicative of the verification error may begenerated. In some embodiments, the verification may be based at leastin part on header and/or tail information as described herein. Asanother example, the receiver 204 may have received the encoded data 206and verified its integrity/authenticity, but has failed to decode thedata properly due to a decoder error. In this case, feedback informationindicative of the decoding error may be generated. In yet anotherexample, the generation of the feedback information may not beresponsive to the receiving of the encoded data, but may be based on thelack of received data. For example, it may be determined that a packetrepresenting at least a portion of a frame is missing based on thesequence numbers of the packets already received. In some embodiments,the generation of the feedback information may be triggered by otherevents such as a hardware/software failure, a lapse of a timer (e.g., apredetermined amount of time has lapsed since the last time feedbackinformation is generated), a change in receiver environment, a userinput, and the like.

In some embodiments, the feedback information can indicate whether anerror has occurred at the receiver and optionally a type of the error.Additionally, the feedback information may include context informationat the receiver, such as an identifier of the last frame or any framebefore the last frame that was successfully decoded at the receiver.Such context information, combined with the status indicator, can beused by the transmitter to customize the encoding of the next frame toimprove the reliability and efficiency of data transmission. Forexample, when there is an error at the receiver, context information canbe useful for determining a suitable error recovery mechanism at thetransmitter, so as to minimize bandwidth and/or changes in bitrate whilesubstantially maintaining the quality of the transmitted bitstream.

The feedback information can be transmitted 210 to the transmitter. Thefeedback information can be transmitted at fixed or varied timeintervals. The intervals may be mutually agreed upon by the transmitterand the receiver. The intervals may be dynamically determined based onvarious factors such as channel capacity, the requirement of atransmitter application regarding the promptness of error recovery,error rate (e.g., bit error rate or bit error ratio) within apredetermined period of time, data regarding previous datatransmissions, and the like.

The transmitter can be configured to determine 212 a mode for encoding acurrent frame based at least in part on the feedback information fromthe receiver. If multiple feedback messages have been received from thereceiver since the last frame encoding, then the latest feedback messagemay be used to determine the coding mode. Determining the coding modemay include selecting a coding mode from a plurality of coding modescomprising an intraframe mode, a short-term interframe mode, and along-term interframe mode. In some examples, determining the coding modemay include selecting a reference frame to be used for interframeencoding.

In some examples, the coding mode may be determined based on a statusindicator included in the feedback information. For example, if an errorhas occurred, then an coding mode associated with a predetermined errorrecovery mechanism may be used. In some cases, additional informationsuch as the type of the error and other information may also be used todetermine the coding mode. If no error has occurred, then a defaultcoding mode (e.g., short-term interframe mode) may be used. Additionallyor alternatively, the mode may be determined based on a differencebetween a frame identifier included in the feedback information and theframe identifier of the current frame. The difference may be comparedwith a predetermined threshold and the comparison result may be used todetermine the coding mode.

In some embodiments, other factors can also be used to determine thecoding mode, such as characteristics of the communication channelbetween the transmitter and the receiver (e.g., capacity, noise, errorrate), use case or application-specific requirement at the transmitterand/or receiver, a state at the receiver, a state at the transmitter,and the like.

Once the coding mode is determined, the current frame can be encoded 214according to the coding mode. The encoded frame data can be transmitted216 to the receiver 204 in one or more packets. Optionally, in someembodiments, the encoded data may be packaged with additional metadatainformation (e.g., header and/or tail information) prior to beingtransmitted to the receiver 204. Such metadata information mayfacilitate efficient verification and/or decoding at the receiver 204.In some other embodiments, the encoded data may not be associated withadditional metadata information.

The receiver 204 can be configured to verify 218 the received encodeddata. The verification may be performed based on the metadatainformation associated with the encoded data and/or the encoded dataitself. The verification can include checking the integrity and/orauthenticity of the encoded data. In various embodiments, theverification can occur at any suitable point in time and theverification can be performed for at least a portion of the encodeddata. For example, the verification can occur with respect to at least aportion of the encoded data before the encoded data is decoded by adecoder at the receiver. Alternatively or additionally, the verificationcan occur with respect to at least a portion of the decoded data afterthe decoding.

The receiver 204 can be configured to decode 220 the encoded frame. Thedecoding can occur according to a decoding mode that correspond to thecoding mode that is determined at step 212. For instance, if the frameis encoded under an intraframe mode, then the frame is decoded usingonly information contained within the frame. If the frame is encodedunder a short-term interframe mode, then the short-term frame is usedfor decoding the frame. If the frame is encoded under a long-terminterframe mode, then the long-term frame is used for decoding theframe. In some embodiments, header/tail information associated with theencoded data can be useful for decoding the frame. For instance, theheader/tail information may indicate a type of the encoded frame (e.g.,I-frame or P-frame). The header/tail information may also indicate anindex of a reference frame relative to the encoded frame, so that adecoder can look up the reference frame to decode the interframe encodedframe.

In some embodiments, the decoded frame may be displayed or played backon the same or a different terminal as the receiver 204.

In various embodiments, frames encoded based on the feedback informationcan promote efficient, effective, and reliable error recovery at thereceiver (e.g., from data loss or decoding error) while minimizing theperformance impact of such error recovery, e.g., on the latency and/orquality of video transmission. The extra context information provided bythe header/tail associated with the encoded data can further improve theoverall efficiency of the system.

FIG. 3 illustrates an exemplary process 300 for data transmission, inaccordance with embodiments. A frame 302 can be encoded 303, by anencoder, to derive encoded data 304, which may comprise a series ofbits. In some embodiments, encoding the frame can include compressing,encryption, or otherwise transforming the frame data. The encoded data304 may be packaged 306 in one or more packets. Each packet 310 maycomprise at least a portion of the encoded data 304 and optionally,metadata 308 about the encoded data. In some embodiments, the metadata308 also includes an identifier or sequence number for the packet. Thepacket identifier can be used by the receiver to detect missing or outof order packets. The packaged data packet(s) 310 can be transmitted 305over a data communication channel (which may be the same or differentfrom the feedback communication channel) to a receiver.

The receiver may be configured to unpack and verify 312 the receiveddata packets 310. In particular, for each packet 310, the metadata 308Aand 308B can be extracted and used for verifying the integrity of theencoded data 304, detecting any error that may have occurred duringtransmission, and checking reconstructability of the encoded frame. Forinstance, if one of a plurality of packets for the encoded frame ismissing, then the frame cannot be constructed. As another example,errors can be detected by checking an error detection code associatedwith the encoded data. In some embodiments, the verification can occurprior to actually decoding the encoded data. Alternatively oradditionally, the metadata may be useful for verifying validity of thedecoded data after the decoding process.

In some embodiments, the metadata can include header data 308A prefixedbefore the encoded data 304 and/or tail data 308B appended after theencoded data 304. For example, the header 308A can include fields suchas a frame identifier of the encoded frame, a type of the encoded frame(e.g., I-frame, P-frame, B-frame), a reference frame offset (e.g., fromthe current frame identifier), a timestamp (e.g., for when the encodingis done), a frame rate (e.g., frames per second), a data rate or bitrate(e.g., megabits per second), a resolution, and the like. The tail 308Bcan include fields such as an error detection code or an errorcorrection code of the encoded frame, a length of the frame, a tailidentifier, and the like. Examples of error detection or correctioncodes can include repetition codes, parity bits, checksums, cyclicredundancy checks (CRCs), cryptographic functions, and the like.

In some embodiments, only the header 308A or the tail 308B may beavailable. In some embodiments, the header 308A and the tail 308B mayinclude more or less information than described above. For example, theheader and/or tail may include a digital signature from a sender of theencoded data and/or a desired feedback transmission interval.

In some embodiments, the error-detection code and frame length includedin the metadata can be used to check the data integrity of the encodeddata (e.g., no data corruption or data loss). The frame identifierincluded in the metadata can be used to ensure it is a new frame thathas not been previously received by the receiver, to detect out-of-orderframes, and/or to detect “skipped” frames. The frame type can be used todetermine whether additional data is required for the reconstruction ofthe frame. For example, an I-frame does not require any additional datato decode the frame, whereas a P-frame requires one or more referenceframes. The reference frame offset can be used to identify the referenceframe required to decode a P-frame. During a check forreconstructability of the encoded frame, the reference frame offset canbe used to determine whether the reference frame is available at thereceiver and whether itself has been reconstructed properly. Thereceived frame can only be reconstructed if both of the above conditionsare satisfied.

In some embodiments, the timestamp can be used by the receiver tocalculate the time interval between the encoding of the frame at thetransmitter and the receipt of the frame at the receiver. The timeinterval may be useful for determining a condition of the communicationchannel, a validity of the received data, a feedback transmissioninterval, and the like. The determination may be based on a comparisonbetween the time interval and one or more predetermined thresholds. Theframe rate and the data rate may be used to determine a suitabledecoding and/or playback strategy. In some embodiments, a digitalsignature included in the metadata can be used to authenticate that theencoded data comes from a trustworthy source. In some embodiments, adesired feedback transmission interval included in the metadata can beused by the receiver to determine a suitable feedback transmissioninterval.

After verification, the encoded data 304 can be decoded 314 by a decoderthat corresponds to the encoder used to generate the encoded data 304,and the decoded data can be used to reconstruct the frame 316. In someembodiments, decoding can include decompressing, decryption, orotherwise reconstructing the frame data. Reconstruction can comprisecombining decoded data in a predetermined manner.

In an embodiment, the metadata described herein can be provided outsideof (e.g., in a separate data structure from) the bitstream generated byan encoder. In an alternative embodiment, the metadata may be providedas part the bitstream generated by the encoder. For example, some codingformats according to certain coding standards include a portion forstoring such user-defined metadata, in addition to a portion for storingthe encoded data. The metadata described herein may be stored as part ofsuch user-defined metadata of the bitstream. The metadata can beextracted by the receiver before decoding of the encoded data, asdescribed elsewhere. Advantageously, in latter embodiment, datatransmitted by can be handled by a standard-compliant decoder (whichsupports a coding format with such a user-defined metadata portion),without requiring non-standard handling of metadata outside thebitstream. In yet some other embodiments, the some portion of themetadata may be placed outside the bitstream while some other portion ofthe metadata may be placed within the bitstream.

In some embodiments, the placement of the metadata (within the bitstreamor outside the bitstream) may be configurable by a user or a systemadministrator. In some example, the placement of the metadata may beautomatically determined based on a type of the codec used toencode/decode the frames. If the codec supports user-defined metadata,then the metadata is transmitted as part of the bitstream. Otherwise,the metadata data is transmitted outside the bitstream.

Certain coding (e.g., compression/decompression) standards limit thetypes of reference frames that may be used for encoding/decoding. Forexample, a certain coding standard may require that a P-frame can onlybe encoded using a short-term reference frame, and does not allow theuse of a long-term reference frame. The techniques described hereinallows encoding and decoding using an arbitrary reference frame, therebyobviating the need for support by the coding standards. To this end, thefeedback information from a receiver can specify a desirable referenceframe to use for encoding a frame. And the metadata from the transmittercan describe the reference frame to use for decoding the frame.

FIG. 4 illustrates exemplary processes 400A and 400B for receiving data,in accordance with embodiments. In some embodiments, the processes maybe implemented by a receiver such as receiver 104 or 204 described inFIG. 1 or FIG. 2, respectively.

Turning first to process 400A. At block 402, a receiving status isdetermined. A receiving status can indicate a status with respect to thereceipt of one or more frames from the transmitter. The receiving statuscan indicate whether encoded data representing an encoded frame has beenreceived successfully (e.g., without loss or corruption of data) and/orwhether the encoded data has been decoded successfully. Thedetermination of the receiving status can be obtained by verification ofthe received data and/or the decoding result. For example, data loss maybe detected when a certain frame is “skipped” and not received (e.g.,frames 1, 2 and 4 are received, but frame 3 is not). As another example,data loss can occur when a packet encapsulating a portion of the encodedframe is not received. As another example, data corruption may bedetected by validating a checksum or CRC of the received data. In someembodiments, the receiving status may indicate a status of ahardware/software component of the receiver, a change in an environmentof the receiver (e.g., increased noise level or interference), a changein physical position/orientation of the receiver, a user input, and thelike. For example, it may be determined whether a hardware device orsoftware application for receiving, decoding, displaying, or playingback of the decoded frame is functioning properly. In variousembodiments, the receiving status may be determined by tracking receiveddata and/or analyzing output or status code from various software and/orhardware components.

At block 404, feedback information is generated based on the receivingstatus. Some or all of the information included in the feedbackinformation can be used by the transmitter to determine how to encodeone or more frames, as described elsewhere herein. In some embodiments,the feedback information can be generated by including a statusindicator and an identifier of a previously-received frame. The statusindicator can indicate the receiving status determined at block 402. Thestatus indicator may indicate a transmission status (e.g., whether thereis data loss or data modification during transmission of a frame) and/ora decoding status (e.g., whether there is an error or failure during thedecoding of a frame). For example, the status indicator can be selectedfrom a plurality of predefined values, each representing a uniquereceiving status. In an example, the status indicator may be set to “0”to indicate that an error occurred and “1” to indicate that no erroroccurred; or vice versa. In another example, the value of the statusindicator may indicate more detailed information about a type of error.For example, the status indicator may be set to “2” to indicate adecoder error and “3” to indicate data corruption or data loss.

The feedback information can also include an identifier of apreviously-received frame. The previously-received frame may be the lastframe or any frame before the last frame that was received and/orreconstructed successfully by the receiver before generation of thefeedback information. In some embodiments, the reference frameidentifier in the feedback information can be set to a predeterminedvalue (e.g., null) to indicate that the receiver does not have areliably reconstructed reference frame and that an intraframe encodedframe (I-frame) is requested. In some embodiments, the status indicatorand the frame identifier are both set in the feedback informationregardless of the status indicator. In some other embodiments, the frameidentifier is only set when the status indicator indicates error.

In some embodiments, the feedback information may include more than astatus indicator and a frame identifier. For example, the feedbackinformation can include more than one frame identifier, such as anidentifier of a successfully reconstructed frame, as well as theidentifier(s) of reference frame(s) used to decode the frame. In somecases, the feedback information can include an indication of whethersome or all of the identified reference frame(s) are still available atthe receiver. As another example, the feedback information can include atimestamp that indicates certain event that occurred at the receiver.For example, the timestamp can indicate a time when a previous frame wasreceived, decoded, or reconstructed at the receiver. As another example,the feedback information may include a feedback interval that indicatesthe current frequency at which feedback information is transmitted.

As an example, assume frames 1 and 2 have been received in sequence andreconstructed properly, but frame 3 is not received after apredetermined period of time has expired since the receipt of the lastframe (“1”, “2” and “3” are frame identifiers for the respectiveframes). Then the feedback information, generated after the expirationof the predetermined period of time, may include a status indicatorindicating that an error (or more specifically, a transmission error)and the frame identifier “2”, which indicates the last frame that hasbeen received and reconstructed properly. Alternatively, the frameidentifier can be any frame that has been successfully received andreconstructed, such as “1”. Similar feedback information may begenerated when frame 3 is “skipped” at the receiver. For example, whenframes 1, 2, and 4 are received by the receiver but not frame 3 is notreceived before frame 4 or not received within a predetermined period oftime from the receiving of frame 2. In cases where an encoded frame istransmitted in more than one packets, each containing a portion of theencoded data, receiving the frame means receiving one or more suchpackets. Correctly or successfully receiving a frame means that allpackets containing the encoded data for the frame are receivedcorrectly.

Without the frame identifier in the feedback information, thetransmitter would need to intraframe encode a subsequent frame as anerror recovery mechanism, to ensure that the receiver can decode theframe without relying on any reference frames. Based on the frameidentifier in the feedback information, the transmitter can insteadinterframe encode a subsequent frame using a reference frame identifiedin the feedback information (e.g., frame 1 or frame 2) and the receiverwould be able to reliably decode the encoded P-frame. Since interframeencoding is typically more efficient than intraframe, the overallefficiency of the system is improved.

At block 406, the feedback information is transmitted to the transmitterat certain intervals. Typically, the shorter the feedback transmissioninterval, the more likely that a certain reference frame (e.g., along-term reference frames or a short-term reference frame) exists in areference frame list at the transmitter, and the more likely an errorrecovery mechanism using a long-term or short-term P-frame will succeed.Also, the shorter the feedback transmission interval is, the closer thereference frame is to the current frame temporally, the more efficientthe encoding will be, and the speedier the recovery. However, morefrequent feedback messages may take up more channel bandwidth,decreasing the efficiency of channel usage.

Conversely, the longer the feedback transmission interval, the morelikely that a certain reference frame (especially a long-term referenceframe) no longer exists in a reference frame list at the transmitter,and the more likely an error recovery mechanism using an I-frame willsucceed. Also, the longer the feedback transmission interval is, thefarther the reference frame is to the current frame temporally, the lessefficient the encoding will be, and the longer it takes to recover fromthe error. However, less frequent feedback messages may take up lesschannel bandwidth, increasing the efficiency of channel usage.

In some embodiments, the feedback information is transmitted at fixedintervals. The fixed intervals may be preset by the system or configuredby a user. In some other embodiments, the feedback information istransmitted at variable intervals. The intervals may be determined (oradjusted) dynamically or in real-time while the transmitter and thereceiver are communicating with each other. In some cases, the intervalmay be determined by the receiver. In some other cases, the interval maybe determined by the transmitter and communicated to the receiver. Insome cases, the interval may be determined by the receiver based oninformation provided by the transmitter. In some embodiments, the fixedor variable intervals may be determined based on a channel capacity, arequirement of a transmitter or receiver application regarding thepromptness of error recovery, a bitrate or an error rate, statisticsregarding previous data transmissions, a state of the transmitter or thereceiver (e.g., attitude, velocity, altitude, position), and the like.

In some embodiments, the feedback transmission intervals may bedetermined or adjusted based on a current condition of the communicationchannel used for transmitting the feedback information (feedbackchannel), such as its capacity and/or quality. For example, when thechannel capacity is greater than or equal to a predetermined threshold,the interval may be lengthened. When the channel capacity is less thanor equal to the predetermined threshold, the interval may be shortened.As another example, when the channel quality is good (e.g., less noiseor interference, strong signal strength), then the interval can belengthened to reduce the burden on the channel. When the channel qualityis poor, the interval can be shortened to ensure more reliable datatransmission.

In some embodiments, the feedback transmission intervals may bedetermined or adjusted based on specific requirements of a use case orapplication scenario regarding timeliness of error recovery. In someexamples, the feedback transmission intervals may be determined oradjusted automatically based on the statistics regarding previous datatransmission including success rate or ratio, error rate (e.g., BER orPER), and the like. For example, the current use case (e.g., based on anetwork environment or hardware configuration of a device used toplayback the reconstructed frames) may require a certain minimum successrate of transmission. For example, it may be required that at least 3frames out of every 5 frames are transmitted correctly. Thus, thepredetermined success rate is ⅗. Then, during the actual transmission offrames, if the average rate of successfully transmitted frames is equalto or greater than ⅗, then the interval may be lengthened. On the otherhand, if the average rate of successfully transmitted frames is lessthan ⅗, then the interval may be lengthened. Similarly, the feedbackinterval may be determined or adjusted based at least in part on anerror rate requirement such as bit error rate or ratio (BER) or packeterror rate or ratio (PER) within a given unit of time. Specifically, theinterval may be shortened when the error rate is equal to or exceeds apredetermined threshold value that is required by a specific usescenario, and lengthened when the error rate is less than thepredetermined threshold value.

In some embodiments, the feedback transmission intervals may bedetermined or adjusted based on statistics related topreviously-transmitted feedback messages. For example, if the number orratio of consecutive feedback messages that indicate error (alsoreferred to as “error messages”) within a predefined period of time isless than a predetermined threshold number or ratio, and/or if the totalnumber or ratio of error messages within a predefined period of time isless than a predetermined threshold number, then the feedback intervalmay be lengthened. On the other hand, if the number or ratio ofconsecutive error messages within a predefined period of time is equalto or greater than a predetermined threshold number or ratio, and/or ifthe total number or ratio of error messages within a predefined periodof time is equal to or greater than a predetermined threshold number,then the feedback interval may be shortened.

In some embodiments, a state of the transmitter, the receiver, oranother device may be used to determine the feedback transmissioninterval. Such state information can include one, two, orthree-dimensional information related to position, angle, velocity,linear acceleration, angular acceleration, and the like associated withthe transmitter (e.g., UAV, camera) or the receiver (e.g., remoteterminal). For instance, the feedback transmission interval can beadjusted by comparing the state information with one or morepredetermined threshold values. As an example, when a velocity (angularor linear) associated with a camera or a UAV reaches or exceeds apredetermined threshold, then the difference (e.g., motion) betweenconsecutive images may be large, and the feedback interval may beshortened. When the velocity (angular or linear) associated with acamera or a UAV is less than a predetermined threshold, then thedifference between consecutive images may be small, and the feedbackinterval may be lengthened.

In some embodiments, the state information may be determined based onone or more sensors associated. The sensors can include, for example,gyroscopes, magnetometers, inertial measurement unit (IMU), visionsensors, GPS sensors, altimeters, and the like.

In some embodiments, the feedback transmission interval can be mutuallyagreed upon by the transmitter and the receiver. For instance, thereceiver can be configured to determine a first interval. The firstinterval may be determined based on any of the factors discussed herein(e.g., channel condition, use case requirement, receiver state). Thereceiver can be configured to receive a second interval from thetransmitter. The second interval may be determined based on any of thefactors discussed herein (e.g., channel condition, use case requirement,transmitter state). Based on the first interval and the second interval,the receiver may be configured to determine the third interval at whichto transmit feedback information. For example, the third interval may bethe greater, the lesser, or the average of the first interval and thesecond interval.

In some examples, the transmission interval may be set to cover theestimated time for a feedback message to reach the transmitter from thereceiver, plus the estimated time for the transmitter to encode the nextframe based on the feedback message, and plus the estimated time for thenext encoded frame to reach the receiver from the transmitter.

In some embodiments, the receiver context information may be optionallyupdated. For example, the context information may include informationrelated to the feedback information such as feedback transmissioninterval, current feedback information, previously-transmitted feedbackinformation, and the like. The receiver context information may alsoinclude information useful for decoding frames such as one or morereference frames that have been decoded successfully. Managing thecontext information may include managing a reference frame list. Asliding window mechanism may be used to control a size of the list. Forinstance, when a new frame is decoded, the reference frame list may beupdated to include newly decoded frame. When the size exceeds apredetermined maximum list size, older frames may be removed from thelist. In some embodiments, a long-term reference frame may be keptwithin the list (e.g., at the head of the list) as long as the error isnot recovered, so that the long-term reference frame can be used todecode an error recovery frame sent by the transmitter. In variousembodiments, the receiver context information may be updated before,during, or after the generation or transmission of the feedbackinformation.

In some embodiments, the receiver may be configured to repeat theprocess 400. For example, after a feedback interval has lapsed since thetransmission of the last feedback message, or if a triggering event hasoccurred, a receiving status may be determined 402 again. Depending onthe new receiving status, a new feedback message may be generated. Forexample, if the first feedback message indicates an error, and duringthe interval between the feedback messages, one or more frames have beenreceived and decoded properly at the receiver, then the second feedbackmessage may be generated to indicate a normal status. Otherwise, if thefirst feedback message indicates an error, and during the intervalbetween the feedback messages, no frames have been received or decodedproperly at the receiver, then the second feedback message may be thesame as the first feedback message. In particular, the frame identifierin the second feedback message may remain the same as the frameidentifier in the first feedback message. In some cases, statusindicator in the first message and the second message may be differentto indicate different types of errors. If the first feedback messageindicates a normal status, and during the interval between the feedbackmessages, an error is detected (e.g., a frame is not received orskipped, or a decoder error is detected), then the second feedbackmessage may be set to include an error indicator and a frame identifierof a previously received frame. If the first feedback message indicatesa normal status, and during the interval between the feedback messages,one or more frames have been received and/or constructed successfully,then the second feedback message may include the same success indicatoras the first feedback message, but the frame identifier may be updatedto identify the latest successfully received/reconstructed frame.

Turning next to process 400B. At block 408, a frame is received by areceiver. At block 410, it is determined whether the received frame canbe successfully reconstructed. The determination can include estimatingthe likelihood of a successful reconstruction, assuming no decodererror. The determination may be made prior to actual decoding of theframe and may be based on metadata associated with the received frame,such as the header/tail information described herein. The determinationcan be made by verifying that a predetermined set of rules aresatisfied. The rules may require that there must be no data loss or datacorruption. Data loss may be detected by checking whether all packetsfor a given encoded frame have been received (e.g., by checking thepacket identifiers and/or frame identifiers of the packets in themetadata). Data corruption may be detected by checking one or more errordetection code or error correction code contained in the metadata.Additionally, different rules may apply depending on the type of thereceived frame (which may be indicated by the metadata). If the receivedframe is an I-frame, then no further check may be necessary, and it isdeemed likely that the I-frame can be reconstructed successfully. If thereceived frame is a P-frame, then a reference frame used to encode theP-frame (which may be indicated by the metadata) may need to be checkedto ensure that (1) the reference frame is available (e.g., in areference frame list maintained by the receiver); and (2) the referenceframe has been successfully received, decoded, and/or reconstructed. Ifboth conditions are satisfied, then the P-frame is deemed that it isdeemed likely that the P-frame can be reconstructed successfully.

In general, the check of reconstructability based on metadata of thereceived data can be performed much quicker and simpler than the actualdecoding of the received data, which can be time consuming andcomputationally complex. Thus, checking for reconstructability of thereceived frame prior to the actual decoding can improve efficiency ofthe system and allow earlier detection of errors and speedier errorrecovery.

If it is determined at block 410 that the received frame can bereconstructed successfully, then a feedback message indicating an OKstatus (also referred to as an “ok feedback message”) can be generatedand transmitted to the receiver. The ok feedback message can include astatus indicator indicating a successful or normal condition. The okfeedback message may or may not include a frame identifier thatidentifies a previously-decoded frame, such as the last successfullydecoded frame or any frame before the last frame.

If it is determined at block 410 that the received frame cannot bereconstructed, then a feedback message indicating an error (alsoreferred to as an “error feedback message”) can be generated andtransmitted to the receiver. The error feedback message can include astatus indicator indicating an error. The status indicator may containthe same error code regardless of the type of error. Alternatively, thestatus indicator can contain different error codes for different typesof errors. For instance, a first error code may be used to indicate anerror detected by the reconstructability check and a second differenterror code can be used to indicate an error detected during the decodingprocess (e.g., a decoder error). Furthermore, the error feedback messagemay or may not include a frame identifier that identifies apreviously-decoded frame, such as the last successfully decoded frame orany frame before the last frame.

At block 414, a received frame is decoded. If there is error duringdecoding, then an error feedback information is provided at block 416.Otherwise, if the received frame is successfully decoded, then theprocess 400B loops back to block 408 to receive the next frame. In someembodiments, decoding includes combining decoded portions of the frame.In such embodiments, decoding and reconstruction can be usedinterchangeably and decoding errors includes an error that occurs duringthe combination of the decoded data. In other embodiments, decoding doesnot include combining decode portions. In such embodiments,reconstruction can include decoding and additional steps for combiningthe decoded data. Feedback messages may or may not be generated inresponse to errors detected during the combination of the decoded data.

FIG. 5 illustrates exemplary processes 500A and 500B for receiving data,in accordance with embodiments. In some embodiments, the processes maybe implemented by a receiver such as receiver 104 or 204 described inFIG. 1 or FIG. 2, respectively. In some embodiments, the steps of theprocesses can be repeated by the receiver.

Turning first to process 500A. At block 502, it is determined whether afirst frame is received correctly. The first frame may be encoded andtransmitted by a transmitter such as transmitter 102 or 202 described inFIGS. 1 and 2, respectively. Determining whether the first frame isreceived correctly includes determining whether there is anytransmission error and/or whether is a decoding error. Transmissionerror can include data loss or data corruption. Data loss may bedetermined by checking whether all packets containing the encode data ofthe frame have been received. Data corruption may be determined bychecking an error detection code and/or an error correction codeassociated with received data. In some examples, a decoder error may beraised by the decoder during decoding of at least a portion of theencoded frame.

At block 504, in response to detecting an error (e.g., transmissionerror or decoding error), a feedback message is transmitted. Thefeedback message can include feedback information indicative of theerror. For example, feedback message can include a status indicator thatindicates the error. Additionally, the feedback message can include aframe identifier that identifies a frame that has been previouslyreceived and decoded properly. For example, the frame identifier canidentify the last frame that was received and decoded properly beforethe transmission of the feedback message. Alternatively, the frameidentifier can identify any other previously decoded frame before thelast frame.

At block 506, a second frame is received after the transmission of thefeedback message. The second frame can be encoded and transmitted by thetransmitter. The second frame may be encoded by the transmitter based onthe feedback information included in the feedback message. For instance,the status indicator and/or the frame identifier in the feedbackinformation may be used to determine whether to intraframe encode orinterframe encode the second frame. The status indicator and/or theframe identifier in the feedback information may be used to determinethe reference frame used to interframe encode the second frame. In someembodiments, the encoding decision may be additionally based on otherfactors such application requirements (e.g., bitrate, frame rate, errorrate), channel capacity, transmitter/receiver state, and the like.

Turning next to process 500B. At block 508, feedback information isgenerated that is indicative of a status at the receiver. The feedbackinformation can indicate generally whether any error has been detectedat the receiver. For example, the feedback information can include astatus indicator with a value of “0” to indicate a normal status and avalue “1” to indicate that an error has occurred. Alternatively oradditionally, the feedback information can indicate whether a specifictype of error has been detected at the receiver. For example, the statusindicator can be “1” to indicate a transmission error, “2” to indicate adecoding error, and “3” to indicate an error associated with asoftware/hardware component. As discussed herein, the feedbackinformation can also include a frame identifier that indicates apreviously and successfully received, decoded and/or reconstructedframe.

At block 510, at least a portion of a correctly transmitted frame isstored in a reference frame list at the receiver. As discussed above,the receiver can be configured to maintain a reference frame list thatis a subset of the reference frame list at the transmitter. Thereference frames in the reference frame list of the receiver can be usedto decode received frames. The reference frame list can be updated toinclude at least a portion of a correctly transmitted and/or decodedframe and to remove older frames in the list, for example, using asliding window mechanism discussed herein.

At block 512, the feedback information is transmitted to the transmitterat a variable interval. As discussed herein, the interval fortransmitting feedback interval can be dynamically adjusted according toa variety of factors such as conditions of the communication channel(e.g., noise or interference), requirements of specific use cases (e.g.,error rate, bitrate, resolution, frame rate, latency, bandwidth),statistics of previously transmitted feedback messages, state oftransmitter and/or receiver, and the like. In some embodiments, theorder of blocks 510 and 512 can be reversed.

FIG. 6 illustrates another exemplary process 600 for receiving data, inaccordance with embodiments. In some embodiments, the process 600 may beimplemented by a receiver such as receiver 104 or 204 described in FIG.1 or FIG. 2, respectively.

At block 604, an error feedback message is generated to request I-frame.This step may be performed when the receiver does not have any correctlyreconstructed frames such as when the receiver initially starts up. Insome embodiments, the receiver may initialize the receiver contextinformation. For example, the reference frame list may be initiallyempty. Since the receiver does not have any reference frame to startwith, an intraframe encoded frame is needed. In some embodiments, thefeedback message may include a status indicator that indicates error.The frame identifier in the feedback message, if any, can be set to nullto indicate that there is no correctly reconstructed reference frame.

At block 606, it is determined whether a new frame is received andwhether it is an intraframe encoded frame, as requested at block 604. Insome embodiments, metadata included with the encoded frame can bechecked that the frame's type is an I-frame (e.g., based on a frame typein the metadata). and that the frame has not been previously received(e.g., based on a frame identifier in the metadata). Such metadata canbe used without decoding the frame to expedite the verification/decodingprocess.

If it is determined at block 606 that a new frame is not an I-frame,then the process 600 loops back to block 604 to request a new I-frame.Similarly, if a new frame is not received within a predetermined amountof time, then the process 600 loops back to block 604 to request anI-frame. Otherwise, if it is determined at block 606 that a new I-frameis received correctly, then the process 600 proceeds to block 608. Insome embodiments, the received I-frame may be verified for dataintegrity using metadata (e.g., CRC, frame length) associated with theencode frame.

At block 608, an ok feedback message is generated and transmitted to thetransmitter. The ok feedback message can include a status indicator ofthe feedback message can be set to OK and the frame identifier can beset to the frame identifier of a previously correctly reconstructedframe, if any. Note that a feedback message is generated and transmittedto the receiver before the received frame is decoded to expedite errorrecovery. The feedback message can be generated based on verification ofmetadata associated with the encoded data without decoding.

At block 608, the received frame is decoded and it is determined whetherthe decoding is successful. If an encoding error is detected, then theprocess 600 loops back to block 604 to generate an error feedbackmessage that indicates the error. Otherwise, the process 600 proceeds toblock 612.

At block 612, it is determined whether a new encoded frame has beenreceived and whether the frame can be reconstructed. Note that as usedhere, the phrase “can be reconstructed” means that the frame is capableof being reconstructed successfully assuming no decoder error. A framethat can be reconstructed may or may not be actually reconstructed. Ifso, then the process 600 loops back to block 608 to generate acorresponding ok feedback message. Otherwise, the process 600 proceedsto block 614.

In some embodiments, determining whether a new frame is received caninclude checking that the frame identifier of a newly received frame(which may be part of the metadata such as a header) has not beenreceived before. In some embodiments, a missing frame can be detected bydetecting a “gap” in the identifiers of recently received frames. Or,the missing frame can be detected when a predetermined amount of timehas expired since when the last frame was received. In some embodimentsdetermining whether a new frame is received includes checking whetherall packets containing encoded data for the new frame have beenreceived.

In some embodiments, determining whether the received frame can bereconstructed can include checking whether the following conditions aresatisfied. First, the frame is received correctly (e.g., no transmissionerror). This may be verified by checking the metadata (e.g., CRC, datalength) associated with the received data. Where the encoded frame istransmitted in multiple packets, all packets must be received withouterror. Second, if the received frame is a long-term P-frame, then thelong-term reference frame used to encode the P-frame must have beenreconstructed successfully and is in the reference frame list at thereceiver. In some embodiments, the metadata associated with the receiveddata may be used to determine the type of the received frame and/or thereference frame used to encode the received frame without decoding thereceived frame. Third, if the received frame is a short-term P-frame,then the short-term reference frame used to encode the P-frame must havebeen reconstructed successfully. The short-term reference frame may ormay not have been added to the reference frame list.

At block 614, if no new frame is received or if a received frame cannotbe reconstructed successfully, then an error feedback message isgenerated and transmitted to the transmitter. For example, a statusindicator of the feedback message can be set to an error code and theframe identifier can be set to the frame identifier of a previouslycorrectly reconstructed frame.

In some embodiments, blocks 612 and 614 can be repeated until a newframe received and can be successfully reconstructed. If not, multipleerror feedback messages may be sent during this time with the same frameidentifier.

FIG. 7 illustrates another exemplary process 700 for receiving data, inaccordance with embodiments. In some embodiments, the process 700 may beimplemented by a receiver such as receiver 104 or 204 described in FIG.1 or FIG. 2, respectively.

At block 704, an error feedback message is generated to request I-frame.This step may be performed when the receiver does not have any correctlyreconstructed frames such as when the receiver initially starts up. Insome embodiments, the receiver may initialize the receiver contextinformation. For example, the reference frame list may be initiallyempty. Since the receiver does not have any reference frame to startwith, an intraframe encoded frame is needed. In some embodiments, thefeedback message may include a status indicator that indicates error.The frame identifier in the feedback message, if any, can be set to nullto indicate that there is no correctly reconstructed reference frame.

At block 706, it is determined whether a new frame is received andwhether it is an intraframe encoded frame, as requested at block 604. Insome embodiments, metadata included with the encoded frame can bechecked that the frame's type is an I-frame (e.g., based on a frame typein the metadata). and that the frame has not been previously received(e.g., based on a frame identifier in the metadata). Such metadata canbe used without decoding the frame to expedite the verification/decodingprocess.

If it is determined at block 706 that a new frame is not an I-frame,then the process 700 loops back to block 704 to request a new I-frame.Similarly, if a new frame is not received within a predetermined amountof time, then the process 700 loops back to block 704 to request anI-frame. Otherwise, if it is determined at block 706 that a new I-frameis received, then the process 700 proceeds to block 708.

At block 708, the received I-frame is decoded and it is determinedwhether reconstruction of the frame based on the decoding is successful.If it is, then the process 700 proceeds to block 710. Otherwise, theprocess 700 proceeds to 704 to request an I-frame.

At block 710, an ok feedback message is generated and transmitted to thetransmitter. The ok feedback message reflects the fact that a new frameis received and reconstructed successfully. For example, a statusindicator of the feedback message can set to OK and the frame identifiercan be set to the frame identifier of the correctly reconstructed frame.

At block 712, it is determined whether a new encoded frame has beenreceived and correctly reconstructed. If so, then the process 700 loopsback to block 710 to transmit a corresponding ok feedback message.Otherwise, the process 700 proceeds to block 714.

At block 714, an error feedback message is generated and transmitted tothe transmitter. The feedback message reflects the fact that a new frameis not received or the new frame is not reconstructed successfully. Forexample, a status indicator of the feedback message can set to an errorcode and the frame identifier can be set to the frame identifier of apreviously correctly reconstructed frame. Blocks 712 and 714 can berepeated until a new frame is successfully decoded. For example,multiple feedback messages may be sent during this time with the sameframe identifier.

In contrast to process 600, where the encoded data is first checked forreconstructability and corresponding feedback messages are sent beforethe received data is actually decoded, here the reconstructability checkis omitted and the encoded data is directly decoded. In some cases,process 700 may be used for receiving data that has no or littlemetadata (e.g., header/tail) useful for data verification beforedecoding.

FIG. 8 illustrates another exemplary process 800 for transmitting data,in accordance with embodiments. In some embodiments, the process 800 maybe implemented by a transmitter such as transmitter 102 or 202 describedin FIG. 1 or FIG. 2, respectively.

At block 802, feedback information from a receiver is received. Forinstance, a feedback message can be received. The feedback message canbe processed to extract feedback information. In various embodiments,feedback information can be transmitted at fixed or variable intervalsas described elsewhere herein. In some embodiments, the feedbackinformation can include a status indicator, which indicates whether anerror (e.g., transmission error, decoding error, software/hardwarefailure) has been detected at the receiver. The value of the statusindicator can indicate an occurrence and/or the type of an error. Thefeedback information can also include a frame identifier that identifiesa frame that has been correctly received and/or decoded at the receiver.

At block 804, a current frame is obtained. For example, the currentframe can be obtained from an image acquisition device or system (e.g.,a camera) configured to capture one or more video frames or stillimages. The image acquisition system may be carried by a movable objectsuch as UAV.

At block 806, the current frame is encoded based at least in part thefeedback information. In some embodiments, a suitable error recoverymechanism is selected based on the feedback information. An errorrecovery mechanism can be used to prevent error in one frame topropagate to subsequent frames. For example, an error recovery mechanismcan involve intraframe encoding a subsequent frame or interframe codingthe frame using only a previously correctly received frame, such asidentified by the frame identifier in the feedback information.

In some embodiments, the feedback information can be used to select acoding mode from a plurality of coding modes, for encoding the currentframe. In some embodiments, if the feedback information indicates thatno error is detected, then the current frame can be encoded using areference frame immediately preceding the current frame. That is, thecurrent frame can be encoded under the short-term interframe codingmode. If the feedback information indicates error, and the referenceframe identified in the feedback information is still available in thereference frame list, then the reference frame can be used to interframeencode the current frame. Thus, the current frame is encoded under thelong-term interframe coding mode. Otherwise, if the feedback informationdoes not include a valid frame reference identifier or if the referenceframe is no longer in the reference frame list (e.g., when the referenceframe list is not empty but does not include the reference frame list orwhen the reference frame list is empty), then the current frame can beintraframe encoded. That is, the current frame can be encoded under theintraframe coding mode.

In some embodiments, the encoding process utilizes and/or updates thetransmitter context information such as a reference frame list. FIG. 9illustrates an exemplary process 900 for managing a reference framelist, in accordance with embodiments. Aspects of the process 900 may beimplemented by a transmitter or a receiver. For example, aspects of theprocess 900 may be implemented by a transmitter context managementmodule 120 of the transmitter 102 or a receiver context managementmodule 134 of the receiver 104 described in FIG. 1.

The reference frame list can include previously frames that have beensuccessfully encoded (e.g., for a transmitter) or decoded (e.g., for areceiver). The reference frame list can be managed using afirst-in-first-out (FIFO) sliding window mechanism. The maximum windowsize may be predetermined (e.g., 2, 4, 8, 10, 16, 24, 32). The minimumwindow size may be 0 or a non-zero number. The window size can beconfigurable by a user or a system administrator. In some cases, thewindow size may be dynamically adjusted, e.g., based on the currentchannel condition, specific use case requirement, and the like. In theillustrated embodiment, new frames are added to the head of the list onthe right and old frames are removed from the tail of the list on theleft. Thus, by default, a short-term reference frame of a current frameis always at the head of the list. In some examples, the index for thehead of the list is 0 and the indices for subsequent elements of thelist increase. In some embodiments, the head of the list is used forinterframe encoding a frame (e.g., by an encoder) or decoding a frame(e.g., by a decoder). Since the short-term reference frame is typicallythe most similar to the current frame, coding efficiency is maximized.However, under certain circumstances, a long-term reference frame may bemoved to the head of the list to encode or decode a current frame whenit is determined that the intervening frames may not have been reliablyreceived.

A short-term reference frame of a current frame is the frame thatimmediately precedes the current frame, temporally. For example, in asystem where the frames are assigned monotonically increasing frameidentifiers as time goes on with an increment of 1, the short-termreference frame for frame N is frame N−1 (where N is an integer greaterthan 0). And the short-term reference frame for frame N+1 is frame N,and so on. A long-term reference frame is any frame temporally precedingthe short-term reference frame. For example, frames N−2 and N−3 can belong-term reference frames for frame N.

As illustrated in FIG. 9, in the reference frame list 902A, frame N−1 isthe short-term reference frame (also referred to as “ST ref frame”)906A, and frame N−3 is a long-term reference frame (also referred to as“LT ref frame”) 908A for the current frame N 904A. Thus, the short-termreference frame N−1 is at the head of the reference frame list (e.g.,with an index of 0) and used to encode the current frame N 904A by atransmitter or used to decode the current frame N 904A by a receiver.

If no error feedback is received from the receiver, then the slidingwindow is advanced forward such that in the updated reference frame list902B, frame N 906B becomes the head of the list (e.g., with an index of0) and the short-term reference frame for the next current frame N+1904B. If the advancement of the sliding window causes the size of thelist to exceed the maximum window size, the oldest frame on the list(i.e., the leftmost frame) may be removed. Thus, the short-termreference frame N is used to encode the now current frame N+1 904B bythe transmitter/or used to decode the now current frame N+1 904B by thereceiver.

If error feedback is received from the receiver, then the next currentframe N+2 904C may need to be encoded (by the transmitter) and decoded(by the receiver) using a long-term reference frame N−3 908C that hasbeen received and/or decoded correctly at the receiver. For instance,the frame N−3 may be identified in the feedback message received fromthe receiver. The long-term reference frame N−3 908C may be moved to thehead of the reference frame list 902C, e.g., after the sliding window isadvanced forward to include the frame N+1. Thus, the head of the list,long-term reference frame N−3 (e.g., with an index of 0), is used toencode the current frame N+2 904C by the transmitter or used to decodethe current frame N+2 904C by the receiver. Alternatively, any frame(e.g., frame N−4) that temporally precedes the reference identified bythe feedback message (e.g., frame N−3) may be used as the long-termreference frame.

In some embodiments, the reference frame list may be cleared when acurrent frame needs to be intraframe encoded to generate an I-frame. AnI-frame may be required in various situations as described herein.

Referring back to block 806 of FIG. 8, in some embodiments, the currentframe can be encoded based on a difference, T_(diff) (a non-negativeinteger), between the current frame identifier and a reference frameidentifier indicated in the feedback information and/or based on whetherthe reference frame is in the reference frame list. In some embodiments,if the status indicator of the feedback information indicates an error,and the T_(diff) is greater than or equal to a predetermined threshold,T_(th) (a non-negative integer), and the reference frame is available inthe reference frame list, then reference frame is used to encode thecurrent frame. If the reference frame is a long-term reference frame,then in some implementations, the long term reference frame may be movedto the head of the reference frame list before it is used to encode thecurrent frame, such as described in FIG. 9.

If the status indicator of the feedback information indicates an error,and the T_(diff) is greater than or equal to T_(th), and the referenceframe is not available in the reference frame list, then the currentframe is intraframe encoded to obtain an I-frame. The long-term P framesand I-frames are transmitted in response to receiver error and can beconsidered error recovery frames. The transmission of I-frames and thetransmission of long-term P-frames can be considered different errorrecovery mechanisms. Otherwise, if the feedback indicates success or ifthe T_(diff) is equal or less than T_(th), then the current frame may beinterframe encoded using a short-term reference frame to obtain ashort-term P frame. Compared with error recovery frames, short-terminterframe encoded frames can improve encoding quality and promoteefficient channel usage.

In some cases, such as when the feedback transmission interval isrelatively long, the receiver error may already be recovered, but thefeedback message (reflecting the recovered status) may not be receivedby the transmitter until later on. During this time, the transmitter maybe configured to transmit one or more unnecessary error recovery frames(e.g., I-frames and long-term P frames), reducing the overall efficiencyof the error recovery mechanism of the communication system. Comparedwith short-term P frames, the error recovery frames typically provideless coding efficiency, encoding quality, and less efficient channelusage. It is therefore desirable to limit the amount of unnecessaryerror recovery frames that are transmitted wherever possible. The use ofthe threshold T_(th) can serve this purpose. T_(th) can represent athreshold time period expressed in the intervening frames. For example,if the current frame rate is 24 frames per second, then if T_(th)=6, thetime interval represented by T_(th) is 0.25 second. In an embodiment,T_(th) can be set to a value that represent an interval that is the sameas or longer than the sum of the current feedback transmission interval(i.e., the interval at which feedback messages are transmitted from thereceiver) and the transmission time or delay for the feedback message.For example, if 6 frames are typically obtained and/or transmitted bythe transmitter during such an interval, then T_(th) may be set to 7 or8. The current feedback transmission interval may be fixed, ordynamically determined by the transmitter and/or the receiver. Forexample, the receiver may inform the transmitter of the current feedbacktransmission interval. In some embodiments, the transmission time usedto determine T_(th) may be calculated based on the transmission time ofone or more recently received feedback messages (e.g., an average, amaximum, a minimum). the transmission time of a feedback message may becalculated based on a timestamp associated with the feedback message.

In some embodiments, the value of T_(th) can be determined based on theintervals between recently received feedback messages. For instance,T_(th) can be set to be larger than an average, a maximum, or a minimumof the intervals between two or more most recently received feedbackmessages. For example, if on average, 8 frames are obtained ortransmitted by the transmitter between when adjacent feedback messagesare received, then T_(th) may be set to be 8 or a higher value of 9 or10.

The above discussion also applies to alternative embodiments where thevalues for Tam and T_(th) represent actual time (e.g., in seconds,milliseconds, microseconds) instead of number of intervening frames. Insuch embodiments, T_(diff) can be determined by comparing the time thecurrent frame is obtained and the time when the reference frame wasobtained. And the value of T_(th) may be determined to be longer thanthe feedback transmission interval plus transmission delay.

In some embodiments, T_(th) may be set be the same as or a slightlyhigher value than T_(f), the interval (expressed as a number ofintervening frames or time) between arrivals of adjacent feedbackmessages. A first value is slightly higher than a second value if thefirst value no more than 50%, 40%, 30%, 20%, or 10% higher than thesecond value. In some other embodiments, T_(th) may be set be more than50% higher than T_(f). For example, T_(th) may exceed T_(f) by be 60%,80%, 100%, 150%, 200%, 300%, 500%, or more.

Alternatively or additionally, the encoding of the current frame can bebased on a condition or an environment at the transmitter. For instance,an I-frame may be required in various situations. When an I-frame isrequired, the reference frame list may be cleared. For example, anI-frame may be required when there is an encoder error, such as streambuffer overflow, encoder hardware failure, and the like. As anotherexample, an I-frame may be required when there is a change in resolutionor bitrate at the encoder. In some embodiments, the transmitter can beconfigured to adjust the resolution and/or bitrate of the transmittedframes to match a current condition of the communication channel and/ora use case. For example, when the channel capacity is abundant, thetransmitter may be configured to transmit frames with higher resolutionand bitrate. Conversely, when the channel capacity is scarce, thetransmitter may be configured to transmit frames with lower resolutionand bitrate. As another example, when a receiver application requires ahigher resolution, the transmitter may be configured to transmit frameswith higher resolution and bitrate; and vice versa. In some embodiments,a rate controller such as the rate controller 118 of FIG. 1 may beconfigured to adjust one or more coding parameters (e.g., quantizationparameter) for the encoder to achieve the desired resolution/bitrate.

At block 808, the encode frame is transmitted to the receiver. In someembodiments, metadata about the encoded data useful for verificationand/or decoding can be packaged with the encoded data prior totransmission.

FIG. 10 illustrates another exemplary process 1000 for transmittingdata, in accordance with embodiments. In some embodiments, the process1000 may be implemented by a transmitter such as transmitter 102 or 202described in FIG. 1 or FIG. 2, respectively.

At block 1002, the transmitter is initialized. The initialization of thetransmitter can include initializing the reference frame list (e.g.,empty). The transmitter can be configured to initialize after startingup and/or after in response to a feedback message from the receiver. Thefeedback message may be similar to the one transmitted at block 604 ofFIG. 6 or block 704 of FIG. 7.

At block 1004, the next frame to be encoded is obtained. The next framemay be a frame from of a stream of images produced by an imageacquisition module such as a video camera or camera. Additionally, astatus of an encoder and/or a status of the receiver can be determined.The encoder status can indicate whether there is an error with theencoder or whether the encoder is functioning properly. The receiverstatus may be determined based on a previously-received feedback messagefrom the receiver.

At block 1006, it is determined whether there is an encoder error or abitrate change. As discussed herein, in some embodiments, thetransmitter can be configured to adjust the bitrate/resolution of thetransmitted image stream based on a condition of the communicationchannel and/or requirements for specific scenarios. If it is true thatthere is an encoder error or a bitrate change, then the process 1000proceeds to block 1008, where the reference frame list is emptied and anintraframe coding mode is selected for encoding the current frame.Otherwise, the process 1000 proceeds to block 1010.

At block 1010, it is determined whether the receiver status is ok orwhether the difference between a current frame identifier and areference frame, T_(diff), is less than the predetermined thresholdvalue, T_(th). The receiver status can be determined based on a feedbackmessage received from the receiver. T_(th) may be determined using basedon any of the various factors described elsewhere herein.

If the condition at block 1010 is satisfied, then the process 1000proceeds to block 1012, where the reference frame list is updated and ashort-term interframe mode is selected for encoding the current frame.The reference frame may be updated using a sliding window mechanism,such that the short-term reference frame is used for encoding thecurrent frame, for example, by inserting the short-term reference frameat the head of the reference frame list. As discussed elsewhere herein,the default coding mode can be set to short-term interframe coding whenthere is no receiver error, so as to improve efficiency associated withthe encoding process and with channel usage. Additionally, the moreefficient short-term interframe coding mode can be used by defaultbetween adjacent feedback messages to avoid transmission of redundanterror recovery frames.

If the condition at block 1010 is not satisfied, then the process 1000proceeds to block 1014, where it is determined whether there is areceiver error and whether a long-term reference frame exists in thereference frame list at the transmitter. The receiver error can beindicated by a feedback message and the long-term reference frame may beidentified in the feedback message. The long-term reference frameidentifier can be used to determine whether it still exists in thereceiver reference frame list, which may be updated using a slidingwindow mechanism.

If the condition at block 1014 is satisfied, then the process 1000proceeds to block 1016, where a long-term interframe coding mode isselected for encoding the current frame. Thus, the error recovery frameis a long-term P frame when the long-term reference frame is available.Such P-frames can be encoded more efficiently (e.g., less bitrate andchannel bandwidth) than I-frames, the other type of error recoveryframes.

If the condition at block 1014 is not satisfied, then the process 1000proceeds to block 1018, where an intraframe coding mode is selected forencoding the current frame. Thus, in some examples, the transmission ofI-frames is limited to only those occasions when long-term andshort-term P frames cannot be used and I-frames are required. Thisimproves coding efficiency and channel utilization.

At block 1020, the current frame is encoded according to the coding modeselected above. Additionally, the encoded frame can be transmitted andthe encoder status can be updated, for example, to reflect whether thereis encoder error. Blocks 1004 to 1020 may be repeated for eachsubsequent frame.

Variation of the embodiments described herein is also within the scopeof the present disclosure. For example, instead of intraframe encoding aframe to generate one I-frame, a plurality of hybrid frames can begenerated wherein each of hybrid frames intraframe encodes a differentportion of the original frame. FIG. 11 illustrates exemplary processesfor encoding a frame 1100, in accordance with embodiments. Under atraditional intraframe coding mode, the entire frame 1100 can be encodedto produce an I-frame 1104. The advantage of intraframe encoding theentire frame is that once the I-frame is received, it can be used toreconstruct the entire frame without relying on any other frames.However, the bits or bitrate required to intraframe encode the entireframe can be much higher than that required for encoding P-frames.Therefore, the transmission of I-frames can cause the undesirablebitrate fluctuation, higher peak-to-average ratio, and longer latency.

An alternative approach can be used to alleviate some of the problems ofusing I-frames. The frame to be encoded 1100 can be divided into aplurality of different portions 1102A, 1102B, 1102C. It is understoodthat the portions shown are for illustrative purposes only and notintended to be limiting. In other embodiments, the frame may comprisemore or less than three portions. A plurality of hybrid frames can begenerated comprising a hybrid frame 1106A, 1106B, 1106C respectivelycorresponding to the portions 1102A, 1102B, and 1102C of the originalframe 1100. The hybrid frames can form a group 1108 that correspond tothe original frame 1100. Each hybrid frame within the hybrid frame groupcontains an intraframe encoded portion (shaded) which is the result ofintraframe encoding a corresponding portion of the original frame. Theremainder of the hybrid frame may comprise the result of interframeencoding the rest of the original frame. Using the hybrid frame 1106A asan example, hybrid frame 1106A comprises an intraframe encoded portion(shaded) resulting from intraframe encoding portion 1102A. Hybrid frame1106B comprises an intraframe encoded portion (shaded area) resultingfrom intraframe encoding portion 1102B. And hybrid frame 1106C comprisesan intraframe encoded portion (shaded area) resulting from intraframeencoding portion 1102C.

Since only a portion of the original frame is intraframe encoded andremainder is interframe encoded, the bitrate required to encode thehybrid frame is lower than that required to encode an I-frame.Therefore, problems of bitrate fluctuation, high peak-to-average ratio,and long transmission delays can be alleviated compared the transmissionof I-frames. Such an approach may be especially suitable for low-latencysystems. However, the drawbacks of this alternative approach is that allhybrid frames within the hybrid frame group need to be receivedcorrectly before the original frame can be successfully reconstructed,thereby prolonging the error recovery process.

In various embodiments, a hybrid frame group can be used in places wherean I-frame is used. For example, on the transmitter side, instead ofgenerating and transmitting an I-frame (e.g., when there is an encodererror, when there is a bitrate change, when the long-term referenceframe does not exist in the reference frame), the transmitter can beconfigured to generate and transmit a group of hybrid frames. Each ofthe hybrid frames can be associated with metadata similar to themetadata for an I-frame, except the type of the frame is a hybrid frameinstead of an I-frame. The metadata may also indicate the intraframeencoded portion in the encoded hybrid frame and/or how the relationshipbetween the hybrid frames within a hybrid frame group. On the receiverside, instead of verifying that an I-frame is correctly received and/ordecoded, the receiver can be configured to verify that all hybrid frameswithin a hybrid frame group are received correctly. Instead of decodingan I-frame, the decoder can be configured to decode the hybrid frameswithin the hybrid frame group to reconstruct the original frame.

The systems, devices, and methods described herein can be applied to awide variety of movable objects. As previously mentioned, anydescription herein of an aerial vehicle, such as a UAV, may apply to andbe used for any movable object. Any description herein of an aerialvehicle may apply specifically to UAVs. A movable object of the presentdisclosure can be configured to move within any suitable environment,such as in air (e.g., a fixed-wing aircraft, a rotary-wing aircraft, oran aircraft having neither fixed wings nor rotary wings), in water(e.g., a ship or a submarine), on ground (e.g., a motor vehicle, such asa car, truck, bus, van, motorcycle, bicycle; a movable structure orframe such as a stick, fishing pole; or a train), under the ground(e.g., a subway), in space (e.g., a spaceplane, a satellite, or aprobe), or any combination of these environments. The movable object canbe a vehicle, such as a vehicle described elsewhere herein. In someembodiments, the movable object can be carried by a living subject, ortake off from a living subject, such as a human or an animal. Suitableanimals can include avines, canines, felines, equines, bovines, ovines,porcines, delphines, rodents, or insects.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof. Alternatively, the movable object may be carried bya living being.

In some instances, the movable object can be an aerial vehicle. Forexample, aerial vehicles may be fixed-wing aircraft (e.g., airplane,gliders), rotary-wing aircraft (e.g., helicopters, rotorcraft), aircrafthaving both fixed wings and rotary wings, or aircraft having neither(e.g., blimps, hot air balloons). An aerial vehicle can beself-propelled, such as self-propelled through the air. A self-propelledaerial vehicle can utilize a propulsion system, such as a propulsionsystem including one or more engines, motors, wheels, axles, magnets,rotors, propellers, blades, nozzles, or any suitable combinationthereof. In some instances, the propulsion system can be used to enablethe movable object to take off from a surface, land on a surface,maintain its current position and/or orientation (e.g., hover), changeorientation, and/or change position.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. The movableobject may be controlled remotely via an occupant within a separatevehicle. In some embodiments, the movable object is an unmanned movableobject, such as a UAV. An unmanned movable object, such as a UAV, maynot have an occupant onboard the movable object. The movable object canbe controlled by a human or an autonomous control system (e.g., acomputer control system), or any suitable combination thereof. Themovable object can be an autonomous or semi-autonomous robot, such as arobot configured with an artificial intelligence.

The movable object can have any suitable size and/or dimensions. In someembodiments, the movable object may be of a size and/or dimensions tohave a human occupant within or on the vehicle. Alternatively, themovable object may be of size and/or dimensions smaller than thatcapable of having a human occupant within or on the vehicle. The movableobject may be of a size and/or dimensions suitable for being lifted orcarried by a human. Alternatively, the movable object may be larger thana size and/or dimensions suitable for being lifted or carried by ahuman. In some instances, the movable object may have a maximumdimension (e.g., length, width, height, diameter, diagonal) of less thanor equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Themaximum dimension may be greater than or equal to about: 2 cm, 5 cm, 10cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance betweenshafts of opposite rotors of the movable object may be less than orequal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.Alternatively, the distance between shafts of opposite rotors may begreater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m,or 10 m.

In some embodiments, the movable object may have a volume of less than100 cm×100 cm×100 cm, less than 50 cm×50 cm×30 cm, or less than 5 cm×5cm×3 cm. The total volume of the movable object may be less than orequal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40 cm³, 50cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³, 300 cm³,500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³3, 1 m³, or10 m³. Conversely, the total volume of the movable object may be greaterthan or equal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40cm³, 50 cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³,300 cm³, 500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³,1 m³, or 10 m³.

In some embodiments, the movable object may have a footprint (which mayrefer to the lateral cross-sectional area encompassed by the movableobject) less than or equal to about: 32,000 cm², 20,000 cm², 10,000 cm²,1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm². Conversely, thefootprint may be greater than or equal to about: 32,000 cm², 20,000 cm²,10,000 cm², 1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm².

In some instances, the movable object may weigh no more than 1000 kg.The weight of the movable object may be less than or equal to about:1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg,8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg,or 0.01 kg. Conversely, the weight may be greater than or equal toabout: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1kg, 0.05 kg, or 0.01 kg.

In some embodiments, a movable object may be small relative to a loadcarried by the movable object. The load may include a payload and/or acarrier, as described in further detail elsewhere herein. In someexamples, a ratio of a movable object weight to a load weight may begreater than, less than, or equal to about 1:1. In some instances, aratio of a movable object weight to a load weight may be greater than,less than, or equal to about 1:1. Optionally, a ratio of a carrierweight to a load weight may be greater than, less than, or equal toabout 1:1. When desired, the ratio of an movable object weight to a loadweight may be less than or equal to: 1:2, 1:3, 1:4, 1:5, 1:10, or evenless. Conversely, the ratio of a movable object weight to a load weightcan also be greater than or equal to: 2:1, 3:1, 4:1, 5:1, 10:1, or evengreater.

In some embodiments, the movable object may have low energy consumption.For example, the movable object may use less than about: 5 W/h, 4 W/h, 3W/h, 2 W/h, 1 W/h, or less. In some instances, a carrier of the movableobject may have low energy consumption. For example, the carrier may useless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less. Optionally,a payload of the movable object may have low energy consumption, such asless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less.

The UAV can include a propulsion system having four rotors. Any numberof rotors may be provided (e.g., one, two, three, four, five, six, ormore). The rotors, rotor assemblies, or other propulsion systems of theunmanned aerial vehicle may enable the unmanned aerial vehicle tohover/maintain position, change orientation, and/or change location. Thedistance between shafts of opposite rotors can be any suitable length.For example, the length can be less than or equal to 2 m, or less thanequal to 5 m. In some embodiments, the length can be within a range from40 cm to 1 m, from 10 cm to 2 m, or from 5 cm to 5 m. Any descriptionherein of a UAV may apply to a movable object, such as a movable objectof a different type, and vice versa.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for a movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject). The load can include a payload and/or a carrier, as describedelsewhere herein.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 12 illustrates a movable object 1200 including a carrier 1202 and apayload 1204, in accordance with embodiments. Although the movableobject 1200 is depicted as an aircraft, this depiction is not intendedto be limiting, and any suitable type of movable object can be used, aspreviously described herein. One of skill in the art would appreciatethat any of the embodiments described herein in the context of aircraftsystems can be applied to any suitable movable object (e.g., an UAV). Insome instances, the payload 1204 may be provided on the movable object1200 without requiring the carrier 1202. The movable object 1200 mayinclude propulsion mechanisms 1206, a sensing system 1208, and acommunication system 1210.

The propulsion mechanisms 1206 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. The movable object may have one or more, two ormore, three or more, or four or more propulsion mechanisms. Thepropulsion mechanisms may all be of the same type. Alternatively, one ormore propulsion mechanisms can be different types of propulsionmechanisms. The propulsion mechanisms 1206 can be mounted on the movableobject 1200 using any suitable means, such as a support element (e.g., adrive shaft) as described elsewhere herein. The propulsion mechanisms1206 can be mounted on any suitable portion of the movable object 1200,such on the top, bottom, front, back, sides, or suitable combinationsthereof.

In some embodiments, the propulsion mechanisms 1206 can enable themovable object 1200 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1200 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1206 can be operable to permit the movableobject 1200 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1206 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1206 can be configured to becontrolled simultaneously. For example, the movable object 1200 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1200. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1200 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1208 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1200 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude global positioning system (GPS) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1208 can be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1200(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1208 can be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1210 enables communication with terminal 1212having a communication system 1214 via wireless signals 1216. Thecommunication systems 1210, 1214 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication; such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1200 transmitting data to theterminal 1212, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1210 to one or morereceivers of the communication system 1214, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1200 and the terminal 1212. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1210 to one or more receivers of the communication system 1214,and vice-versa.

In some embodiments, the terminal 1212 can provide control data to oneor more of the movable object 1200, carrier 1202, and payload 1204 andreceive information from one or more of the movable object 1200, carrier1202, and payload 1204 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1206), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1202).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1208 or of the payload 1204). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload. The controldata provided transmitted by the terminal 1212 can be configured tocontrol a state of one or more of the movable object 1200, carrier 1202,or payload 1204. Alternatively or in combination, the carrier 1202 andpayload 1204 can also each include a communication module configured tocommunicate with terminal 1212, such that the terminal can communicatewith and control each of the movable object 1200, carrier 1202, andpayload 1204 independently.

In some embodiments, the movable object 1200 can be configured tocommunicate with another remote device in addition to the terminal 1212,or instead of the terminal 1212. The terminal 1212 may also beconfigured to communicate with another remote device as well as themovable object 1200. For example, the movable object 1200 and/orterminal 1212 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1200, receivedata from the movable object 1200, transmit data to the terminal 1212,and/or receive data from the terminal 1212. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1200 and/orterminal 1212 can be uploaded to a website or server.

FIG. 13 is a schematic illustration by way of block diagram of a system1300 for controlling a movable object, in accordance with embodiments.The system 1300 can be used in combination with any suitable embodimentof the systems, devices, and methods disclosed herein. The system 1300can include a sensing module 1302, processing unit 1304, non-transitorycomputer readable medium 1306, control module 1308, and communicationmodule 1310.

The sensing module 1302 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1302 can beoperatively coupled to a processing unit 1304 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1312 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1312 canbe used to transmit images captured by a camera of the sensing module1302 to a remote terminal.

The processing unit 1304 can have one or more processors, such as aprogrammable or non-programmable processor (e.g., a central processingunit (CPU), a microprocessor, an FPGA, an application-specificintegrated circuit (ASIC)). The processing unit 1304 can be operativelycoupled to a non-transitory computer readable medium 1306. Thenon-transitory computer readable medium 1306 can store logic, code,and/or program instructions executable by the processing unit 1304 forperforming one or more steps. The non-transitory computer readablemedium can include one or more memory units (e.g., removable media orexternal storage such as an SD card or random access memory (RAM)). Insome embodiments, data from the sensing module 1302 can be directlyconveyed to and stored within the memory units of the non-transitorycomputer readable medium 1306. The memory units of the non-transitorycomputer readable medium 1306 can store logic, code and/or programinstructions executable by the processing unit 1304 to perform anysuitable embodiment of the methods described herein. The memory unitscan store sensing data from the sensing module to be processed by theprocessing unit 1304. In some embodiments, the memory units of thenon-transitory computer readable medium 1306 can be used to store theprocessing results produced by the processing unit 1304.

In some embodiments, the processing unit 1304 can be operatively coupledto a control module 1308 configured to control a state of the movableobject. For example, the control module 1308 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1308 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1304 can be operatively coupled to a communicationmodule 1310 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1310 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1310 can transmit and/or receive one or more of sensing data from thesensing module 1302, processing results produced by the processing unit1304, predetermined control data, user commands from a terminal orremote controller, and the like.

The components of the system 1300 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1300 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 13 depicts asingle processing unit 1304 and a single non-transitory computerreadable medium 1306, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1300 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1300 can occur at one or more of theaforementioned locations.

While some embodiments of the present disclosure have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the disclosure. It should beunderstood that various alternatives to the embodiments of thedisclosure described herein may be employed in practicing thedisclosure. It is intended that the following claims define the scope ofthe invention and that methods and structures within the scope of theseclaims and their equivalents be covered thereby.

What is claimed is:
 1. A computer-implemented method, comprising:receiving, from a receiver of image data, a feedback message comprisinga reference frame identifier, the reference frame identifier beingassociated with a reference frame correctly received or decoded by thereceiver; determining a difference between the reference frameidentifier and a current frame identifier, the current frame identifierbeing associated with a current frame to be encoded and the differenceindicating a number of frames between the reference frame and thecurrent frame; and encoding the current frame based at least in part onthe difference.
 2. The method of claim 1, wherein the encoding of thecurrent frame comprises: encoding the current frame based at least inpart on a comparison between the difference and a predeterminedthreshold, wherein the predetermined threshold is equal to or greaterthan a number of frames between adjacent feedback messages indicating aninterval between the adjacent feedback messages.
 3. The method of claim2, wherein the difference or the predetermined threshold is anon-negative integer.
 4. The method of claim 2, wherein the encoding ofthe current frame further comprises: selecting a coding mode from aplurality of coding modes based at least in part on the comparison, theplurality of coding modes comprising an intraframe coding mode, ashort-term interframe coding mode, and a long-term interframe codingmode.
 5. The method of claim 2, wherein the encoding of the currentframe comprises: selecting a short-term interframe coding mode to encodethe current frame in response to the difference being less than thepredetermined threshold; or selecting a long-term interframe coding modeor an intraframe coding mode to encode the current frame in response tothe difference being greater than or equal to the predeterminedthreshold.
 6. The method of claim 1, further comprising: transmittingthe encoded current frame to the receiver.
 7. The method of claim 6,further comprising: packaging the encoded current frame with metadataprior to transmitting the encoded current frame to the receiver.
 8. Themethod of claim 7, wherein the metadata comprises at least one of anidentifier of the current frame, a type of the encoded current frame, areference frame offset, a timestamp, a frame rate, a data rate, aresolution, verification data, or an error-detection code.
 9. The methodof claim 1, wherein the feedback message further comprises a statusindicator indicating a receiving status, a transmission status, or adecoding status with respect to the image data, or indicating amalfunction of a hardware or software component of the receiver.
 10. Themethod of claim 9, wherein the encoding of the current frame comprises:selecting a coding mode from a plurality of coding modes based at leastin part on the status indicator.
 11. The method of claim 1, wherein theencoding of the current frame comprises interframe encoding the currentframe using the reference frame.
 12. The method of claim 1, wherein thefeedback message further comprises a timestamp associated withgeneration or transmission of the feedback message, an interval at whichadjacent feedback messages are being sent, or synchronizationinformation for data transmission.
 13. A computer system, comprising: amemory that stores one or more computer-executable instructions; and oneor more processors configured to access the memory and execute thecomputer-executable instructions to perform a method comprising:receiving, from a receiver of image data, a feedback message comprisinga reference frame identifier, the reference frame identifier beingassociated with a reference frame correctly received or decoded by thereceiver; determining a difference between the reference frameidentifier and a current frame identifier, the current frame identifierbeing associated with a current frame to be encoded and the differenceindicating a number of frames between the reference frame and thecurrent frame; and encoding the current frame based at least in part onthe difference.
 14. The computer system of claim 13, wherein theencoding of the current frame comprises: encoding the current framebased at least in part on a comparison between the difference and apredetermined threshold, wherein the predetermined threshold is equal toor greater than a number of frames between adjacent feedback messagesindicating an interval between the adjacent feedback messages.
 15. Thecomputer system of claim 14, wherein the encoding of the current framefurther comprises: selecting a coding mode from a plurality of codingmodes based at least in part on the comparison, the plurality of codingmodes comprising an intraframe coding mode, a short-term interframecoding mode, and a long-term interframe coding mode.
 16. The computersystem of claim 14, wherein the encoding of the current frame furthercomprises: selecting a short-term interframe coding mode to encode thecurrent frame in response to the difference being less than thepredetermined threshold; or selecting a long-term interframe coding modeor an intraframe coding mode to encode the current frame in response tothe difference being greater than or equal to the predeterminedthreshold.
 17. The computer system of claim 13, wherein the methodfurther comprises: packaging the encoded current frame with metadata,wherein the metadata comprises at least one of an identifier of thecurrent frame, a type of the encoded current frame, a reference frameoffset, a timestamp, a frame rate, a data rate, a resolution,verification data, or an error-detection code; and transmitting thepackaged encoded current frame to the receiver.
 18. The computer systemof claim 13, wherein the feedback message further comprises a statusindicator indicating a receiving status, a transmission status, or adecoding status with respect to the image data, or indicating amalfunction of a hardware or software component of the receiver.
 19. Thecomputer system of claim 18, wherein the encoding of the current framecomprises: selecting a coding mode from a plurality of coding modesbased at least in part on the status indicator.
 20. One or morenon-transitory computer-readable storage media storing computerexecutable instructions that, when executed by a computing system,configure the computing system to perform a method comprising:receiving, from a receiver of image data, a feedback message comprisinga reference frame identifier, the reference frame identifier beingassociated with a reference frame correctly received or decoded by thereceiver; determining a difference between the reference frameidentifier and a current frame identifier, the current frame identifierbeing associated with a current frame to be encoded and the differenceindicating a number of frames between the reference frame and thecurrent frame; and encoding the current frame based at least in part onthe difference.